PHILOSOPHY OF DATA SCIENCE 2: Natural Language Processing

Summer 2023 Syllabus

Instructor : Ignacio Ojea Quintana.

Meeting Time: Mondays 12 to 14
Location: Ludwigstr. 31 - 021 (floor plan)
Office Hours: By appointment
Contact: ignacio.ojea@lmu.de
Uni Link: Link
Google Drive Link : Link

Course Description

This course will focus on the philosophy of computational language models.

On the one hand, we will cover the fundamentals of Natural Language Processing (NLP). The goal is
to provide the student skills and techniques for them to do research using them, and also to make our
philosophical discussion informed about the methods in the field. These techniques include: pre-processing,
vectorizing, embedding, principal component analysis, logistic and naive bayes classification, topic modeling,
neural networks, and finally attention models. It is preferable if the student has some experience with Python
and linear algebra.
On the other hand, the focus on NLP is motivated by the success of recent models like (Chat-)GPT
(4), Bard, Bert, etc. Are these models teaching us something new about the nature of language? What are
the central philosophical questions about this new technology? NLP allows us to think about the transition
from symbolic to non-symbolic (or statistical) Artificial Intelligence, since language is at the core of the AI
project. In particular, we will consider these model’s (lack of) ability to abstract hierarchical, syntactical
and/or logical structure from natural language, and whether they have the capacity for inference in the way
we humans do.
To finish, the course will include a tutorial at the beginning of each of the lectures in order to cover the
programming skills component of it.

Prerequisites

The course will require the students to do NLP themselves on different corpus of data. For this reason,
some coding experience is welcomed. Interest in the philosophy of language and mind is a substantial bonus.

Required Texts

• All texts will be made available digitally.

Grading

There will be two components to the evaluation, namely writing a philosophical essay and an NLP project.
In order to help students build up to the final project, optional exercises will be uploaded throughout
the course.

TENTATIVE COURSE SCHEDULE

1
 Please note that all readings and due dates are subject to change.
Please do the readings before attending to class.
Note: Dates will be corrected according to the academic calendar. This course is about
twelve weeks long, factoring in public holidays.

Part 1: Language and Symbolic AI

Week 1: Introduction: The Turing Test Mon 17/04

Required:
• Please come with your computer with Anaconda installed as well as packages: numpy,
pandas, random, matplotlib, nltk, PyPDF2, re.
• A. Turing, “Computing Machinery and Intelligence”, [19].

• Tutorial 1: Python basics.

Optional:
• J.H. Moor, “An Analysis of the Turing Test”, [11].
• M. Halina, “Insightful artificial intelligence”, [9].

Week 2: The Language of Thought Mon 24/04

Required:

• J. Fodor, “Propositional Attitudes”, [6].

• S. Schneider, “The Language of Thought”, [17].

• Tutorial 2: Linear algebra basics like dot product, matrix transpose, etc.

Week 3: HOLIDAY: Labour Day Mon 01/05

HOLIDAY: Labour Day

Week 4: Mind as Software (guest speaker: Giorgio Sbardolini) Mon 08/05

Required:

• N. Block, “The Mind as the Software of the Brain”, [2]

• J. Searle, “Can Computers Think?”, [18].

• Tutorials 3: Vectorization, Feature Selection, Bag of Words Representation.

Optional:
• M. Boden, “Escaping from the Chinese Room”, [3].
• D. Dennett, “Can Machines Think?”, [5].

2
Week 5: Technical Meeting: More Linear Algebra Mon 15/05

Required:
• Make sure you are up to date with the tutorial exercises and content.
• Think of linear algebra as the underlying grammar of the models we will be using.
• We will provide notes on these concepts: eigenvectors, eigenvalues, singular value decomposition,
euclidean distance, cosine similarity. They are relatively basic, so feel free to prepare using any resources
you find useful.
• Tutorial 4: Linear Algebra.

Week 6: Wrapping Up Symbolic AI Mon 22/05

Required:
• D. Marr, “The Philosophy and the Approach”, in [10].
• Optional presentation of any of the optional texts.
• Tutorial 5: Principal Component Analysis and Topic Modeling.
Optional:
• E. Bender and A. Kollner, ”Climbing towards NLU: On Meaning, Form, and Understanding in the
Age of Data”, [1]
• J.H. Moor, “An Analysis of the Turing Test”, [11].
• M. Halina, “Insightful artificial intelligence”, [9].
• M. Boden, “Escaping from the Chinese Room”, [3].
• D. Dennett, “Can Machines Think?”, [5].

Part 2: Connectionism and Statitistical AI

Week 7: HOLIDAY: Pentecost Monday Wed 29/05

HOLIDAY: Pentecost Monday

Week 8: Probabilistic Language of Thought Mon 05/06

Required:
• S. Piantadosi and R. Jacobs, “Four Problems Solved by the Probabilistic Language of Thought”, [15].
• S. Piantadosi, “Modern language models refute Chomsky’s approach to language”.
• Tutorial 6: Hidden Markov Models.
Optional:
• R. Katzir, “Why large language models are poor theories of human linguistic cognition. A reply to
Piantadosi (2023)”, link.
• Noam Chomsky, “The False Promise of ChatGPT”, NYTs article.

3
• N. Goodman, J. Tenenbaum, T. Gerstenberg, “Concepts in a Probabilistic Language of Thought”, [8].

Week 9: Getting into deep learning Mon 12/06

Required:
• J. Fodor and Z. Pylyshyn, “Connectionism and Cognitive Architecture: A Critical Analysis”, [7].
• J. Pater, “Generative linguistics and neural networks at 60: foundations, frictions, and fusion”, [13].
• Tutorial 7: Word Embeddings.

Week 10: Do LLMs have understanding? Mon 19/06

Required:

• E. Bender and A. Koller, “Climbing towards NLU: On Meaning, Form, and Understanding in the Age
of Data”, [1].
• Reply paper

• Tutorial 8: Neural Networks for Sentiment Analysis.

Week 11: OpenAI’s Main Papers Mon 26/06

Required:

• OpenAI Team, “Language Models are Few-Shot Learners”, [4].

• OpenAI Team, “Training language models to follow instructions with human feedback”, [12].
• Tutorial 9: Next Word Prediction, N-Grams.

Optional:

• OpenAI Team, “Improving Language Understanding by Generative Pre-Training”, [16].

Week 12: Technical Session: Attention Models Mon 03/07

Required:

• Google Team, “Attention Is All You Need”, [20].

• There are plenty of online tutorials to also complement this session.

• Tutorial 10: Attention Models.

Week 13: Inference In Language Models Mon 10/07

Required:

• TBU
• TBU

4
 • Tutorial 11: Review Session.

Optional:

• TBU

Week 14: Open Session: General Discussion, papers suggested by students Mon 17/07

Required:

• Pavlik, “Semantic Structure in Deep Learning”, [14]

• Tutorial 12: Review Session.

Examination Week:

Final Project Due

Final Paper Due

References
[1] Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding
in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 5185–5198, Online, July 2020. Association for Computational Linguistics.

[2] Ned Block. The mind as the software of the brain. In Daniel N. Osherson, Lila Gleitman, Stephen M.
Kosslyn, S. Smith, and Saadya Sternberg, editors, An Invitation to Cognitive Science, Second Edition,
Volume 3, pages 377–425. Cambridge MA: MIT Press, 1995.
[3] Margaret A. Boden. Escaping from the chinese room. In John Heil, editor, Computer Models of Mind.
Cambridge University Press, 1988.

[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-
Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario
Amodei. Language models are few-shot learners. CoRR, abs/2005.14165, 2020.
[5] Daniel C. Dennett. Can Machines Think?, pages 295–316. Springer Berlin Heidelberg, Berlin, Heidel-
berg, 2004.
[6] Jerry A. Fodor. Propositional attitudes. The Monist, 61(October):501–23, 1978.

[7] Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis.
Cognition, 28(1):3–71, 1988.
[8] Noah D. Goodman, Joshua B. Tenenbaum, and Tobias Gerstenberg. Concepts in a probabilistic language
of thought. 2014.
[9] Marta Halina. Insightful artificial intelligence. Mind & Language, 36, 01 2021.

[10] David Marr. Vision. W. H. Freeman, 1982.

[11] James H. Moor. An analysis of the turing test. Philosophical Studies: An International Journal for
Philosophy in the Analytic Tradition, 30(4):249–257, 1976.

5
[12] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton,
Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan
Lowe. Training language models to follow instructions with human feedback, 2022.
[13] Joe Pater. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language,
95:e41 – e74, 2019.
[14] Ellie Pavlick. Semantic structure in deep learning. Annual Review of Linguistics, 8(1):447–471, 2022.
[15] Steven T. Piantadosi and Robert A. Jacobs. Four problems solved by the probabilistic language of
thought. Current Directions in Psychological Science, 25:54 – 59, 2016.

[16] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training.
2018.
[17] Susan Schneider. The language of thought. In John Symons and Paco Calvo, editors, Routledge Com-
panion to Philosophy of Psychology. Routledge, 2009.

[18] John R. Searle. Can computers think? In David J. Chalmers, editor, Philosophy of Mind: Classical
and Contemporary Readings. Oup Usa, 2002.
[19] A. M. Turing. Computing machinery and intelligence. Mind, 59(236):433–460, 1950.
[20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.