reveal.js

"Compute" as a Noun

Computational Power and AI

Jürgen Stowasser

January 2024

Compute as a Noun

Computational power (CPU, GPU, IoT) as basic requirement for large-scale generative AI
Building, Training, Inference, Deployment
Extremely large datasets
Compute as commodity (Global Chip Shortage 2020-?) & geopolitical instrument (US "CHIPS and Science Act", 2022)

"We're currently training our next-gen model Llama 3, and we're building massive compute infrastructure to support our future roadmap, including 350k H100s by the end of this year -- and overall almost 600k H100s equivalents of compute if you include other GPUs."
	Marc Zuckerberg announcing the development of the open-source large language model *Llama 3*, 18.01.2024
	[https://www.instagram.com/reel/C2QARHJR1sZ/](https://www.instagram.com/reel/C2QARHJR1sZ/).
    ---
    “Right now in AI, what worries me is we don’t have the resources to make sure that academic AI continues to be a centre of gravity. Because if we lose that centre of gravity, then the other centre of gravity is driven by capitalism, Public-sector investment in AI is so abysmal. Not a single university today can train a ChatGPT model . Academia cannot fully develop its own versions so that it can be used for more open scientific research. That is a problem.”

Fei-Fei Li, Human-Centered AI Institute at Stanford (‘Maths is pretty clean. Humans are messy’, *Financial Times* Dec 15, 2023)

*The term "artificial intelligence" means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to—
(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an automated manner; and
(C) use model inference to formulate options for information or action..*
	[National Artificial Intelligence Initiative Act of 2020](https://www.congress.gov/bill/116th-congress/house-bill/6216)
    ---
    *AI is the ability of a machine to display human-like capabilities such as reasoning, learning, planning and creativity.
AI enables technical systems to perceive their environment, deal with what they perceive, solve problems and act to achieve a specific goal. The computer receives data - already prepared or gathered through its own sensors such as a camera - processes it and responds.*
[European Parlament, 2020/2023](https://www.europarl.europa.eu/news/en/headlines/society/20200827STO85804/what-is-artificial-intelligence-and-how-is-it-used)
    ---
	*"AI socio-technical systems": Technology is always part of society, just like society is always part of technology. This also means that one cannot understand one without the other. Technology is not only design and material appearance but also sociotechnical; that is, a complex process constituted by diverse social, political, economic, cultural and technological factors.*
[EU-U.S. Terminology and Taxonomy for Artificial Intelligence, 31 May 2023](https://digital-strategy.ec.europa.eu/en/library/eu-us-terminology-and-taxonomy-artificial-intelligence)

ImageNet (Fei-Fei Li et al.)

Inspired by Irving Biederman's "Recognition-by-components: A theory of human image understanding perceptual recognition of objects" (1987)
Generalisation as mathematical argument for the need of large databases
Large image (database) built by the global AI community (and Amazon Mechanical Turk workers)
Parallel project for language: (WordNet) (Sources: Reuters, Decoder)

Large Language Models

"Stochastic Parrots"?
"Sparks of Artificial General Intelligence (AGI)?"
"Spontaneous Emergence of an AI Theory of Mind"?

Sparks of AGI(?)

Datasets & Databases

Training Data for LLMs: Sources???
Bias and Debiasing: Hidden Human Labour
"CCP-approved" dataset (50 billion tokens sized) released by the Cyberspace Administration of China (CAC) (Sources: Reuters, Decoder)

NYT Sues OpenAI and Microsoft

Millions of articles were used to train chatbots that now compete with the NYT
Counts include copyright infringement, unfair competition by misappropriation, trademark dilution
Complaint filed 12/27/23

Fawkes Cloaking

Nightshade (SAND Lab, Univ of Chicago

Nightshade [..] is designed as an offense tool to distort feature representations inside generative AI image models. [..] While human eyes see a shaded image that is largely unchanged from the original, the AI model sees a dramatically different composition in the image.

↓

For example, human eyes might see a shaded image of a cow in a green field largely unchanged, but an AI model might see a large leather purse lying in the grass. Trained on a sufficient number of shaded images that include a cow, a model will become increasingly convinced cows have nice brown leathery handles and smooth side pockets with a zipper.

Tlahcuiloh