Skip to content

Quiz: Tracking AI Progress and Trends

Test your understanding of AI progress measurement, benchmarking, and trend analysis.

Question 1: What is "task horizon" in the context of measuring AI capabilities?

  1. How far in the future AI can make predictions
  2. The length of tasks (in human time) that AI can complete autonomously at a given reliability threshold
  3. The maximum number of tasks AI can handle simultaneously
  4. The time until AI becomes generally intelligent
Answer

B) The length of tasks (in human time) that AI can complete autonomously at a given reliability threshold - Task horizon translates abstract AI capabilities into practical, understandable terms by measuring how long a task takes a skilled human professional.

Question 2: According to METR research, approximately how often do AI task completion capabilities double?

  1. Every 2 years (similar to Moore's Law)
  2. Every 12 months
  3. Every 7 months
  4. Every 3 months
Answer

C) Every 7 months - This is one of the fastest capability growth rates observed in any technology domain, with significant implications for strategic planning.

Question 3: What caused the "Power Wall" phenomenon around 2004?

  1. A global power shortage
  2. Thermal and power consumption limits that prevented further increases in CPU clock speed
  3. A software limitation
  4. Patent restrictions on processor design
Answer

B) Thermal and power consumption limits that prevented further increases in CPU clock speed - This led to a shift toward parallel processing, which coincidentally aligned well with AI workloads and enabled GPU-based deep learning.

Question 4: What does Moore's Law predict about transistor counts?

  1. Transistor counts remain constant over time
  2. The number of transistors on a chip doubles approximately every two years
  3. Transistor counts decrease annually
  4. Transistor growth follows a linear pattern
Answer

B) The number of transistors on a chip doubles approximately every two years - Gordon Moore's 1965 observation predicted exponential growth in transistor density, which has held roughly true for over 50 years and enabled modern AI systems.

Question 5: Why is tracking AI benchmarks important for business strategy?

  1. Benchmarks have no business relevance
  2. It helps time investments, plan workforce changes, and anticipate competitive dynamics
  3. Benchmarks are only useful for AI researchers
  4. Tracking benchmarks is required by law
Answer

B) It helps time investments, plan workforce changes, and anticipate competitive dynamics - Understanding AI capability trajectories enables better strategic planning around adoption timing, workforce development, and competitive positioning.

Question 6: How have AI benchmarks evolved over time?

  1. They have become simpler and easier to pass
  2. They have remained the same since the 1990s
  3. They have progressed from simple pattern recognition to professional-level tasks like coding and legal reasoning
  4. They have been completely replaced by human evaluation
Answer

C) They have progressed from simple pattern recognition to professional-level tasks like coding and legal reasoning - Early benchmarks focused on basic tasks, while modern benchmarks test specialized skills that previously required human expertise.

Question 7: What is the MMLU benchmark designed to measure?

  1. Model memory usage
  2. Knowledge across 57 academic subjects
  3. Image generation quality
  4. Processing speed
Answer

B) Knowledge across 57 academic subjects - The Massive Multitask Language Understanding benchmark tests AI models across a wide range of academic disciplines from science to humanities.

Question 8: If AI task completion capabilities are at 5 hours and continue doubling every 7 months, approximately how long would the task horizon be after 28 months?

  1. 10 hours
  2. 20 hours
  3. 40 hours
  4. 80 hours
Answer

D) 80 hours - With 4 doublings over 28 months (28 ÷ 7 = 4), capabilities would progress: 5 → 10 → 20 → 40 → 80 hours.

Question 9: What is an important caveat when using AI capability projections for planning?

  1. Projections are always accurate
  2. Projections assume current trends continue, but physical limits, economic factors, or algorithmic plateaus could alter the trajectory
  3. Projections should be ignored entirely
  4. Only short-term projections matter
Answer

B) Projections assume current trends continue, but physical limits, economic factors, or algorithmic plateaus could alter the trajectory - While projections are valuable for planning, organizations should prepare for multiple scenarios rather than assuming any single projection is certain.

Question 10: How did the Power Wall's shift to parallel processing benefit AI development?

  1. It had no effect on AI
  2. Parallel processing aligned perfectly with the matrix operations required for neural network training
  3. It made AI development impossible
  4. It only affected video games
Answer

B) Parallel processing aligned perfectly with the matrix operations required for neural network training - The shift from faster single cores to multiple parallel cores enabled GPU-based deep learning, as neural networks require many simultaneous calculations.

Question 11: What does the LM Arena (LMSYS Chatbot Arena) use to rank language models?

  1. Academic test scores only
  2. Human preferences through blind comparisons using an ELO rating system
  3. Processing speed benchmarks
  4. Company market capitalization
Answer

B) Human preferences through blind comparisons using an ELO rating system - Users compare outputs from anonymous models, and the aggregated preferences create ELO rankings similar to chess ratings.

Question 12: Which of the following is NOT typically considered a driver of AI acceleration?

  1. More training data
  2. Better algorithms
  3. Increased government regulation
  4. More compute power
Answer

C) Increased government regulation - While regulation may affect AI deployment, it is not a driver of capability acceleration. The main drivers are data, algorithms, compute, and feedback loops where AI helps develop better AI.

Question 13: According to the Four Futures framework, what two dimensions determine different AI scenarios?

  1. Cost and speed
  2. Pace of AI advancement and distribution of AI benefits
  3. Hardware and software quality
  4. Open source vs. proprietary development
Answer

B) Pace of AI advancement and distribution of AI benefits - The Four Futures framework considers whether AI advances quickly or slowly, and whether benefits are broadly shared or concentrated.

Question 14: What strategic action is recommended based on AI trend analysis?

  1. Wait until AI is perfect before adopting
  2. Ignore AI trends and focus only on current capabilities
  3. Plan for accelerating change and monitor key benchmarks while preparing for multiple scenarios
  4. Assume AI progress will stop soon
Answer

C) Plan for accelerating change and monitor key benchmarks while preparing for multiple scenarios - Organizations should use trend data to inform strategy while acknowledging uncertainty and preparing for different possible futures.

Question 15: What year marked a significant breakthrough for deep learning with AlexNet winning ImageNet?

  1. 2005
  2. 2012
  3. 2017
  4. 2022
Answer

B) 2012 - AlexNet's victory in the ImageNet competition demonstrated the power of deep neural networks trained on GPUs, launching the modern deep learning era.