LLM summarisation should really only ever be considered a starting point and must never be relied upon. Particularly for academic texts where the language complexity varies substantially. Generally you get better results from targeting a specific type of summary in the prompt rather than simply asking ‘summarise this academic paper’. Wherever possible you want to break down tasks into smaller, more manageable chunks to avoid the LLM being overloaded with the length of text and risk data loss or confusion, or deciding what’s important to summarise. Here’s an example giving it the paper Generative AI for Economic Research: Use Cases and Implications for Economists (Korinek 2023) which contains substantial detail on specific use cases and asked it for a broad summary firstly where LLMs can be valuable, and secondly where they are less valuable for economic research:
Prompt:
Attached is an article on the applied use of LLMs for social science research, focusing specifically on economics research. Please provide 2 distinct summaries: the first should focus on the positive valuable applications mentioned in the article, and the second should focus on the problems or limitations mentioned in the article.
Response:
Summary 1: Positive Valuable Applications
The article highlights numerous valuable applications of large language models (LLMs) for economic research:
1. Ideation and feedback: LLMs can help brainstorm research ideas, evaluate hypotheses, and provide counterarguments.
2. Writing assistance: LLMs are useful for synthesising text, editing, evaluating writing style, and generating titles or abstracts.
3. Background research: They can summarise long texts, explain concepts, and assist with literature reviews.
4. Coding: LLMs can write, explain, translate between languages, and debug code.
5. Data analysis: They can create figures, extract data from text, reformat data, classify text, and extract sentiment.
6. Mathematical derivations: LLMs show emerging capabilities in setting up models and explaining mathematical concepts.
7. Productivity gains: Researchers can significantly increase productivity by incorporating LLMs into their workflow, especially for automating micro-tasks.
8. Tutoring: LLMs can act as tutors for coding, data analysis, and explaining economic concepts.
9. Simulating human subjects: LLMs can be used to simulate responses in economic experiments.
10. Future potential: The article suggests that as LLMs continue to improve, they may revolutionise economic research and potentially automate many less advanced or creative cognitive tasks currently performed by human researchers.
Summary 2: Problems and Limitations
The article also highlights several problems and limitations of using LLMs in economic research:
1. Hallucinations: LLMs can produce inaccurate or false information with high confidence, requiring careful human oversight.
2. Limited knowledge cutoff: Most LLMs have a knowledge cutoff date and cannot access real-time information.
3. Bias: LLMs can perpetuate biases present in their training data.
4. Inconsistent performance: Some LLM capabilities are still experimental and produce inconsistent results.
5. Mathematical limitations: Current LLMs struggle with complex mathematical derivations and high-level abstractions.
6. Privacy concerns: LLMs may compromise privacy by generating outputs based on sensitive training data or retaining user inputs.
7. Homogenisation risk: Widespread use of a small set of LLMs could lead to more homogeneous research ideas and approaches.
8. Ethical concerns: The use of LLMs to simulate human subjects raises ethical questions about the validity and representativeness of such data.
9. Overreliance risks: Researchers may become overly dependent on LLMs, potentially reducing their critical thinking skills.
10. Job displacement: In the long term, advanced LLMs could potentially displace human economists in certain tasks or research areas.
11. Need for human oversight: Despite their capabilities, LLMs still require significant human supervision and validation of outputs.
12. Limitations in literature research: LLMs often struggle with accurately citing or referencing academic literature.
Generally speaking, focusing the summary to a particular area produces better quality results than simply ‘Please summarise this paper’, which essentially leaves it up to the LLM to decide what’s valuable, which doesn’t necessarily align with what a researcher needs. An important caveat to note especially for student researchers is the temptation for relying on LLM summaries can be very risky including for fundamental data loss or misleading interpretations, but also on long term critical reading skills. For novel research projects, more often than not the most valuable insights for a specific research project come not from the abstract or headline findings but from some less prominent paragraph or caveat or idea. Reading papers in full with critical engagement will always be required and LLM support with reading should only ever be a way to support your productivity and not something to rely on.