Upload files to "/"

english_explain.jsonl

@@ -0,0 +1,20 @@
+{"type": "english_explain", "instruction": "Explain what the 'brute-force approach' to training Large Language Models refers to, according to the document.", "input": "", "reasoning": "The document contrasts brute-force with a curated paradigm. Brute-force relies on sheer, unfiltered data volume, leading to statistical fluency but functional brittleness.", "output": "The brute-force approach in LLM training means using massive, unfiltered web data without careful selection. It prioritizes quantity over quality, resulting in models that are statistically fluent (generate plausible text) but functionally brittle (fail at complex or subtle language tasks)."},
+{"type": "english_explain", "instruction": "Explain what FineWeb-Edu is and how its educational score filtering works.", "input": "", "reasoning": "FineWeb-Edu is a subset of FineWeb filtered by an automated educational quality classifier using Llama-3-70B-Instruct annotations and a linear regressor on embeddings, retaining only documents with score >=3.", "output": "FineWeb-Edu is a dataset of 5.4 trillion tokens derived from FineWeb (which came from Common Crawl snapshots). It uses an automated educational quality classifier: first, the Llama-3-70B-Instruct model annotates 460,000 candidate texts with an additive score (0-5). Then a linear regressor built on Snowflake-arctic-embed-m embeddings predicts educational scores for all texts. Only documents scoring 3 or higher are kept, ensuring the model trains on structured, academically rigorous content rather than noisy web chatter."},
+{"type": "english_explain", "instruction": "Explain what the Common Corpus is and why it addresses the 'open data paradox'.", "input": "", "reasoning": "Common Corpus provides 2 trillion permissibly licensed tokens from public domain sources with exhaustive provenance documentation, solving the conflict between open access and legal/ethical restrictions.", "output": "The Common Corpus is a 2-trillion-token dataset (2,003,039,184,047 tokens) of permissibly licensed data. It solves the 'open data paradox' — the conflict between wanting open access to data and respecting copyright/ethical constraints — by meticulously sourcing public domain texts (classic literature, historical newspapers, government documents, legal case law) and documenting their provenance exhaustively. This allows LLMs to train legally and ethically while accessing high-quality, diverse English registers."},
+{"type": "english_explain", "instruction": "Explain what JFLEG is and how it differs from traditional grammar datasets.", "input": "", "reasoning": "JFLEG (JHU Fluency-Extended GEC) focuses on holistic fluency edits rather than isolated error correction, forcing the model to rewrite entire sentences for natural flow and native-like rhythm.", "output": "JFLEG (JHU Fluency-Extended GEC) is a gold-standard dataset for Grammatical Error Correction. Unlike traditional grammar datasets that focus only on isolated spelling or tense errors, JFLEG emphasizes holistic fluency edits. It requires the model to rewrite entire sentences so they flow naturally, teaching the LLM the rhythm, cadence, and idiomatic structuring of native-level prose rather than just fixing discrete mistakes."},
+{"type": "english_explain", "instruction": "Explain what CoNLL-U format is and how it helps LLMs understand complex sentence structure.", "input": "", "reasoning": "CoNLL-U provides 10-field annotations for morphology, part-of-speech tagging, and dependency parsing. Training on explicit syntactic trees forces the model to map exact word relationships rather than using shallow heuristics.", "output": "CoNLL-U is a standard format for Universal Dependencies that provides explicit annotations across 10 distinct fields per token, including morphological features, part-of-speech tags, and dependency relations (e.g., subject, object, modifier). Training an LLM on CoNLL-U annotated data forces it to perform step-by-step reasoning over syntactic heads before predicting dependency labels. This prevents the model from relying on shallow pattern matching and instead requires it to map the exact grammatical relationship between every word in a sentence, enabling comprehension of highly complex, multi-clause structures."},
+{"type": "english_explain", "instruction": "Explain what the S2ORC dataset is and why it is important for achieving PhD-level English proficiency.", "input": "", "reasoning": "S2ORC contains structured full text, abstracts, and metadata for over 10 million open-access academic papers. It saturates the model with high-density scientific text and unambiguous terminology for expert-level discourse.", "output": "S2ORC (Semantic Scholar Open Research Corpus) is an unparalleled dataset for academic training, comprising structured full text, abstracts, and metadata for over 10 million open-access English academic papers. It is critically important for PhD-level proficiency because it saturates the model with high-density scientific text — precise, unambiguous terminology, formal argumentation structures, and citation-dense exposition. Training on S2ORC expands the model's vocabulary to include expert-level discourse across disciplines, enabling it to generate publication-ready scientific prose rather than conversational approximations."},
+{"type": "english_explain", "instruction": "Explain what Chain-of-Thought (CoT) prompting is and how the CoT Collection trains LLMs to reason step-by-step.", "input": "", "reasoning": "CoT prompting forces models to generate intermediate reasoning steps before final answers. The CoT Collection augments 1.88 million rationales across 1,060 tasks, instilling deliberate multi-step problem-solving behavior.", "output": "Chain-of-Thought (CoT) prompting is a technique that enables complex reasoning by forcing the model to explicitly generate intermediate reasoning steps before outputting a final conclusion. Rather than jumping directly from question to answer, the model produces a logical chain: 'First, ... Second, ... Therefore, ...'. The CoT Collection is a massive dataset containing 1.88 million CoT rationales across 1,060 distinct tasks. Continually fine-tuning on this collection instills a deliberate, multi-step problem-solving behavior that resists the urge to leap to linguistically coherent but logically flawed conclusions. This significantly enhances zero-shot accuracy on complex benchmarks like BIG-Bench-Hard."},
+{"type": "english_explain", "instruction": "Explain what the Philosophy-instruct dataset is and how it trains LLMs for abstract reasoning and wisdom.", "input": "", "reasoning": "Philosophy-instruct is generated from top philosophical texts on Project Gutenberg, providing 720,000+ trainable tokens of multi-turn instructional dialogue. It teaches ethical dilemmas, Socratic dialogues, and balanced weighing of opposing viewpoints.", "output": "The Philosophy-instruct dataset is generated from the most significant philosophical texts available on Project Gutenberg, providing over 720,000 trainable tokens of multi-turn, instructional dialogue. Through continued pre-training and subsequent instruction tuning on these foundational texts, the LLM learns to navigate ethical dilemmas, construct Socratic dialogues, and weigh opposing viewpoints with intellectual maturity. This targeted exposure enables the model to transcend simple factual recall and provide deeply contemplated, balanced, and profoundly 'wise' answers to ambiguous human queries — precisely the capability that distinguishes a thoughtful scholar from a stochastic parrot."},
+{"type": "english_explain", "instruction": "Explain what the SARC dataset is and why it is indispensable for training sarcasm detection in LLMs.", "input": "", "reasoning": "SARC (Self-Annotated Reddit Corpus) contains 1.3 million sarcastic comments with heavy contextualization within preceding conversational turns. Sarcasm requires contradictory semantic states (literal vs intended meaning) and context beyond sentence boundaries.", "output": "The SARC (Self-Annotated Reddit Corpus) is indispensable for sarcasm detection because it contains 1.3 million sarcastic comments scraped from Reddit, uniquely characterized by their heavy contextualization within preceding conversational turns. Sarcasm presents a unique computational challenge: the model must hold two contradictory semantic states simultaneously — the literal meaning of the text AND the intended, diametrically opposed meaning. Because sarcasm depends heavily on context and shared cultural knowledge, training on SARC forces the LLM to look beyond the immediate sentence boundary to analyze broader conversational history, speaker intent, and rhetorical patterns. Without this training, models interpret sarcastic statements literally, missing the pragmatic meaning entirely."},
+{"type": "english_explain", "instruction": "Explain what the Deceptive Humor Dataset (DHD) is and how it categorizes humor for LLM training.", "input": "", "reasoning": "DHD categorizes humor into five archetypes (Dark Humor, Irony, Social Commentary, Wordplay, Absurdity) and grades each on a Satire Level from 1 (subtle) to 3 (overt). This teaches models to distinguish harmless puns from biting social commentary.", "output": "The Deceptive Humor Dataset (DHD) is a groundbreaking resource that categorizes humor into five specific archetypes: Dark Humor (jokes about taboo or morbid subjects), Irony (expressing meaning opposite to literal), Social Commentary (humor critiquing society or politics), Wordplay (puns and linguistic tricks), and Absurdity (illogical or surreal situations). Crucially, DHD grades each instance on a 'Satire Level' from 1 (subtle satire that may be mistaken for sincerity) to 3 (overt satire that is unmistakably humorous). Training on DHD allows the LLM to differentiate between a harmless phonetic pun and biting, deceptive social commentary that borders on misinformation. This granularity is essential for generating contextually appropriate humor without offending users or spreading deceptive content masquerading as jokes."},
+{"type": "english_explain", "instruction": "Explain what the Surge AI Profanities dataset is and how it categorizes offensive language for LLM training.", "input": "", "reasoning": "Surge AI catalogs over 1,600 English profanities categorized by semantic target (sexual acts, bodily fluids, physical attributes, political offense) and grades each on severity scale 1 (mild) to 3 (severe) based on human consensus.", "output": "The Surge AI Profanities dataset is a comprehensive resource cataloging over 1,600 English profanities and offensive terms. Each term is categorized by its semantic target — the type of offense it represents: sexual acts (e.g., explicit sex terms), bodily fluids (scatological references), physical attributes (body-shaming terms), or political offense (slurs targeting identity groups). More importantly, every term is graded on a severity scale from 1 (mild — casual swearing acceptable among friends) to 3 (severe — highly offensive terms that constitute harassment) based on human consensus ratings. By ingesting this structured data, the LLM learns the precise lexical weight and social consequence of each word, enabling it to distinguish between casual, acceptable banter among friends and severe, targeted harassment that should be filtered or avoided entirely."},
+{"type": "english_explain", "instruction": "Explain the four-stage data curation pipeline proposed in the document's conclusion for achieving world-class English proficiency.", "input": "", "reasoning": "Stage 1: Foundational pre-training with FineWeb-Edu and Common Corpus for pristine vocabulary and formal register. Stage 2: Syntactic mastery with JFLEG, Lang-8, CoNLL-U for flawless grammar and coreference. Stage 3: Cognitive reshaping with CoT Collection, DeepResearch Bench, GRAPE for deep academic reasoning. Stage 4: Pragmatic fluency with Gutenberg, LiteraryTaste, SARC, DHD, Surge AI for wit, humor, and native-like communication.", "output": "The document proposes a four-stage data curation pipeline. Stage 1: Foundational pre-training using FineWeb-Edu (5.4 trillion educationally filtered tokens) and Common Corpus (2 trillion permissibly licensed tokens) to build a pristine vocabulary and formal register while ensuring ethical compliance. Stage 2: Syntactic mastery through Grammatical Error Correction datasets (JFLEG, Lang-8), dependency parsing (CoNLL-U), and bridging anaphora training (GUMBridge) to achieve flawless, multi-clause grammar and paragraph-level cohesion. Stage 3: Cognitive reshaping using Chain-of-Thought Collection (1.88 million rationales), DeepResearch Bench (100 PhD-level tasks), and the GRAPE framework for distribution alignment — forcing the model to abandon shallow heuristics in favor of deep, step-by-step academic reasoning. Stage 4: Pragmatic fluency through literary corpora (Gutenberg, LiteraryTaste), sarcasm detection (SARC, PodSarc), humor understanding (Deceptive Humor Dataset), idiom comprehension (MAGPIE), and profanity mastery (Surge AI, Jigsaw) — elevating the model from a functional machine to a fluent, witty, native-like communicator."},
+{"type": "english_explain", "instruction": "Explain the updated token size and construction pipeline of FineWeb-Edu based on 2025 sources.", "input": "", "reasoning": "Recent sources state FineWeb-Edu contains 1.3 trillion tokens, not 5.4 trillion, and describe extraction, heuristics, deduplication, and Llama-3-70B-Instruct scoring.", "output": "FineWeb-Edu is a 1.3-trillion-token educational subset of the FineWeb corpus, which aggregates text from 96 Common Crawl snapshots. The pipeline includes: (1) extraction via trafilatura for HTML-to-text; (2) heuristic quality filters (terminal punctuation, duplicated lines); (3) per-snapshot MinHash deduplication with 5-grams; and (4) educational quality filtering where Llama-3-70B-Instruct annotates 460,000 candidates, a linear regressor on Snowflake-arctic-embed-m embeddings predicts scores 0–5, and documents scoring ≥3 are retained."},
+{"type": "english_explain", "instruction": "Explain the six principal collections of Common Corpus and their token distributions.", "input": "", "reasoning": "Common Corpus has six collections: Open Government (406B), Open Culture (886B), Open Science (281B), Open Code (283B), Open Web (73B), and Open Semantic (68B).", "output": "Common Corpus comprises six principal collections: (1) Open Government (406 billion tokens) – administrative, legal, and fiscal texts from sources like SEC, WTO, and EU documents; (2) Open Culture (886 billion tokens) – digitized monographs, newspapers, and periodicals across 13+ languages; (3) Open Science (281 billion tokens) – openly licensed scientific publications from OpenAlex, arXiv, theses, and clinical trials; (4) Open Code (283 billion tokens) – permissively licensed code from Stack v1/v2 across 600+ programming languages; (5) Open Web (73 billion tokens) – Wikipedia, Wikisource, StackExchange, and YouTube transcripts; (6) Open Semantic (68 billion tokens) – natural language renditions of Wikidata in 300 languages. Total exceeds 2 trillion tokens across 517+ million documents."},
+{"type": "english_explain", "instruction": "Explain the S2ORC dataset's scale and what structured information it provides beyond raw text.", "input": "", "reasoning": "S2ORC contains 81.1M English-language academic papers with rich metadata, abstracts, resolved bibliographic references, and structured full text for 8.1M open access papers.", "output": "S2ORC (Semantic Scholar Open Research Corpus) is a large corpus of 81.1 million English-language academic papers spanning many academic disciplines. It provides rich metadata (title, authors, year, venue), paper abstracts, resolved bibliographic references linking citations to their source papers, and structured full text for 8.1 million open access papers. The full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. This structure enables research on citation networks, figure/table extraction, and fine-grained scientific text mining."},
+{"type": "english_explain", "instruction": "Explain the CoNLL-U format and its 10-field annotation structure for universal dependencies.", "input": "", "reasoning": "CoNLL-U has 10 tab-separated fields: ID, FORM, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC. It encodes token-level morphological features and dependency relations.", "output": "CoNLL-U (Computational Natural Language Learning – Universal) is a tab-separated format with 10 fields per token: (1) ID – token index; (2) FORM – word form; (3) LEMMA – base form; (4) UPOS – universal part-of-speech; (5) XPOS – language-specific POS; (6) FEATS – morphological features (e.g., Tense=Past, Number=Sing); (7) HEAD – index of syntactic head; (8) DEPREL – dependency relation to head (e.g., nsubj, obj, amod); (9) DEPS – enhanced dependencies; (10) MISC – additional annotations (e.g., bridging anaphora). This structure enables explicit encoding of complete syntactic structure for each sentence."},
+{"type": "english_explain", "instruction": "Explain the core hypothesis and selection mechanism of the GRAPE framework for instruction tuning.", "input": "", "reasoning": "GRAPE hypothesizes SFT is most effective when data aligns with the model's pretrained distribution. It selects the response with highest probability from the target model for each instruction.", "output": "GRAPE (which stands for the framework proposed in 'The Best Instruction-Tuning Data are Those That Fit') is a supervised fine-tuning (SFT) framework based on the hypothesis that SFT is most effective when the data aligns with the model's pretrained distribution. For each instruction, GRAPE gathers multiple responses from various LLMs (or human annotators), computes the probability of each response according to the target model's distribution, and selects the response with the highest probability. Standard SFT is then performed only on these probabilistically aligned responses, producing supervision that is distributionally well-matched to the model rather than imitative of an external teacher."},
+{"type": "english_explain", "instruction": "Explain what The Pile dataset is and its 22 diverse sources according to 2025 documentation.", "input": "", "reasoning": "The Pile is an 825 GB dataset combining 22 sources including PubMed, arXiv, GitHub, and legal documents. It forces models to learn cross-disciplinary vocabularies.", "output": "The Pile is an 825 GB English-language dataset combining 22 diverse, high-quality sources. Key sources include: PubMed Central (biomedical literature), arXiv (physics, math, CS preprints), GitHub (code repositories), USPTO (patents), FreeLaw (legal opinions), and PhilPapers (philosophy). The diversity of domains forces the model's embedding space to map highly specialized, domain-specific vocabularies, improving performance on academic question-answering tasks. The dataset excludes low-quality web text, prioritizing curated, professional sources."},
+{"type": "english_explain", "instruction": "Explain the ProX (Programming Every Example) framework and its sample-wise rule generation approach for data refinement.", "input": "", "reasoning": "ProX treats data refinement as code generation: a 0.3B LLM writes and executes small programs (string normalization, HTML stripping) for each dataset record.", "output": "ProX (Programming Every Example) is a framework that treats data refinement as a code generation task. A lightweight 0.3B-parameter language model writes and executes small, targeted programs for every individual record in a dataset. These programs perform operations such as string normalization, HTML tag stripping, duplicate sentence removal, or regex-based pattern replacement. Unlike fixed heuristics applied uniformly, ProX generates record-specific cleaning logic based on the content of each document. This approach scales to massive datasets like C4 (750 GB), RedPajama-V2, and DCLM, producing a pristine corpus free of typographical noise and formatting artifacts."}
+

english_reasoning.jsonl

@@ -0,0 +1,19 @@
+{"type": "english_reasoning", "instruction": "Why does the author claim that a model trained with brute-force data is 'statistically fluent but functionally brittle'? Provide logical steps based on the text.", "input": "", "reasoning": "Brute-force uses volume over quality → model learns surface patterns but not deep structure → handles common phrases well but fails at sub-surface complexities like sarcasm, long-range dependencies, or academic rigor.", "output": "Step 1: Brute-force training relies on sheer, unfiltered data volume from web crawls. Step 2: Such data contains noise, contradictions, and shallow patterns. Step 3: The model learns statistical correlations (fluency) but not true understanding of syntax, pragmatics, or reasoning. Step 4: When faced with complex English (e.g., philosophical arguments, bridging anaphora, sarcasm), the model produces coherent-sounding but logically or contextually wrong answers (brittleness). Hence, functional brittleness emerges from lack of deep linguistic structure in training."},
+{"type": "english_reasoning", "instruction": "Why does training on educationally filtered web data (FineWeb-Edu) improve MMLU and ARC scores more than raw web data? Provide a logical chain.", "input": "", "reasoning": "Raw web data contains low-quality, unstructured, or factually inconsistent text. Educational filtering retains texts with logical flow, structured exposition, and factual accuracy. MMLU and ARC require knowledge application and reasoning, which benefit from such structure.", "output": "Step 1: Raw web crawls include forums, personal blogs, and low-credibility sources with fragmented or contradictory information. Step 2: Educational filtering (score ≥3) selects texts that exhibit structured exposition, logical flow, and academic rigor. Step 3: Pre-training on such data teaches the model to recognize coherent argumentation and factual consistency. Step 4: MMLU and ARC benchmarks test multi-task understanding and reasoning — tasks that require structured knowledge. Step 5: Hence, the model generalizes better to these benchmarks, producing the observed 12% and 24% gains respectively."},
+{"type": "english_reasoning", "instruction": "Why does the document claim that including legal and financial data from Common Corpus helps the model achieve a 'formal, precise, and professional register'? Provide logical steps.", "input": "", "reasoning": "Legal/financial texts have specific conventions: precise terminology, conditional clauses, unambiguous references. Training on them exposes the model to high-information-density, low-ambiguity language, which it can then generalize to similar formal contexts.", "output": "Step 1: Legal and financial documents are characterized by high information density, precise terminology, conditional logic, and minimal ambiguity. Step 2: The Common Corpus includes 406.6 billion tokens from these domains (case law, patents, financial filings). Step 3: During pre-training, the model's embedding space learns the statistical and syntactic patterns unique to this register. Step 4: When the model later encounters a query requiring formal output (e.g., regulatory analysis, contract interpretation), it can seamlessly adopt that register. Step 5: Thus, exposure to legal/financial data directly enables a formal, precise, professional tone."},
+{"type": "english_reasoning", "instruction": "Why is it important that an LLM understands the difference between native speaker errors and non-native speaker errors? Provide logical reasoning based on the text.", "input": "", "reasoning": "Different error patterns require different correction strategies. Native errors (punctuation, spelling) are often typos or stylistic; non-native errors (determiners, articles) reflect missing grammatical rules. A world-class model must handle both.", "output": "Step 1: Native English speakers typically err on punctuation (e.g., comma splices), spelling (homophones like 'their/there'), and run-on sentences. Step 2: Non-native speakers typically err on determiners (a/an/the), articles, preposition usage, and subject-verb agreement. Step 3: If an LLM only trains on non-native corrections (e.g., learner corpora), it may overcorrect native-like informal writing. Step 4: If it only trains on native corrections, it may fail to understand systematic second-language errors. Step 5: Training on C4_200M (synthetic errors) plus Lang-8 ensures the model is robust against all forms of input degradation, making it capable of correcting both a native speaker's typo and a learner's article mistake."},
+{"type": "english_reasoning", "instruction": "Why does mastery of bridging anaphora serve as the 'definitive dividing line' between a mediocre LLM and an excellent one? Provide logical steps.", "input": "", "reasoning": "Bridging anaphora requires tracking implicit references across sentences. Models without this skill generate repetitive or disjointed text. Models with it produce cohesive, logically connected paragraphs that maintain referential clarity.", "output": "Step 1: Bridging anaphora requires the model to infer unstated relationships between entities (e.g., 'door' belongs to 'house' without explicit restatement). Step 2: A model lacking this skill will either repeat the antecedent ('The house has a door. The house's door is red') or lose coherence entirely. Step 3: This results in disjointed, repetitive, or confusing paragraphs that fail to maintain logical flow. Step 4: A model trained on GUMBridge learns to maintain implicit referential chains across sentences without restating nouns. Step 5: This enables cohesive paragraphs where each sentence logically follows from the previous one. Step 6: Therefore, bridging anaphora mastery is the definitive dividing line between a model that merely generates sentences and one that writes coherent, deeply intertwined prose."},
+{"type": "english_reasoning", "instruction": "Why does expanding answer options from 4 to 10 (as in MMLU-Pro) eliminate the possibility of success through random guessing? Provide probabilistic reasoning.", "input": "", "reasoning": "Random guessing on 4 options gives 25% chance; on 10 options gives 10% chance. More importantly, with 10 plausible distractors, shallow heuristics (e.g., keyword matching) fail because many options share keywords. The model must truly understand the concept.", "output": "Step 1: With 4 answer options, random guessing yields a 25% baseline accuracy. Step 2: With 10 answer options, random guessing drops to 10% baseline. Step 3: However, the elimination is not just about probability — it's about distractor quality. Step 4: With only 3 distractors, it's easier to eliminate wrong answers using surface cues (e.g., length, keyword overlap, common phrases). Step 5: With 9 distractors, many will share surface features, domain vocabulary, and plausible phrasing with the correct answer. Step 6: A model relying on shallow heuristics (e.g., 'match the keywords from the question') will fail because multiple options match. Step 7: Only a model with deep conceptual understanding — not pattern matching — can reliably identify the correct answer among 10 highly plausible distractors. Thus, MMLU-Pro forces genuine graduate-level comprehension."},
+{"type": "english_reasoning", "instruction": "Why does aligning SFT data with the model's pre-trained distribution (GRAPE) produce better results than using more data? Provide logical steps.", "input": "", "reasoning": "Out-of-distribution SFT data causes catastrophic forgetting or degraded robustness. GRAPE selects responses that fit the model's existing knowledge, preserving learned representations while adding new capabilities efficiently.", "output": "Step 1: A model's pre-trained weights represent a specific probability distribution over tokens — what it has learned about language from vast pre-training. Step 2: Traditional SFT often uses instructional data that is out-of-distribution compared to this pre-trained knowledge (e.g., different style, format, or reasoning patterns). Step 3: Out-of-distribution fine-tuning forces the model to override its learned representations, leading to catastrophic forgetting or degraded robustness. Step 4: GRAPE solves this by selecting, for each instruction, the response that best matches the model's existing probability distribution. Step 5: This preserves the model's core knowledge while adding instruction-following capabilities. Step 6: Because the model doesn't have to 'unlearn' or radically adapt, it learns faster and with less data. Step 7: Hence, a smaller model with GRAPE-selected data (1/3 the data, half the epochs) outperforms a baseline trained on 4.5x more unaligned data."},
+{"type": "english_reasoning", "instruction": "Why is training on parallel stylistically distinct versions of the Bible (same content, different style) effective for teaching text style transfer? Provide logical steps.", "input": "", "reasoning": "Style transfer requires changing form while preserving meaning. Parallel Bible versions have identical semantics but wildly different vocabulary/syntax. This creates a perfect paired dataset for learning which transformations preserve meaning and which alter it.", "output": "Step 1: Text Style Transfer (TST) requires the model to change the stylistic form of text (e.g., formal → casual, archaic → modern) while preserving semantic content exactly. Step 2: Most datasets lack ground-truth pairs where meaning is identical but style differs. Step 3: The Bible, however, exists in multiple stylistically distinct English versions (e.g., King James Version: archaic, formal; New International Version: modern, accessible). Step 4: Across these versions, the underlying semantic content (theological narratives, parables, teachings) remains identical. Step 5: This creates a perfect parallel corpus: aligned sentence pairs where meaning is constant but vocabulary, syntax, and register vary widely. Step 6: Training on this data (evaluated via BLEU and PINC metrics) teaches the model the precise mechanics of paraphrasing — which words can be substituted, which syntactic structures can be reorganized, and how to maintain semantic fidelity across radical stylistic shifts. Step 7: The model learns that 'he spake unto them' and 'he said to them' are semantically equivalent but stylistically distinct, enabling sophisticated rhetorical adaptation."},
+{"type": "english_reasoning", "instruction": "Why does sarcasm detection require the model to hold 'two contradictory semantic states simultaneously'? Provide logical reasoning with an example.", "input": "", "reasoning": "Sarcasm communicates the opposite of literal meaning. To understand 'Great weather we're having' during a hurricane, the model must simultaneously process literal meaning (positive weather statement) and intended meaning (negative complaint), then recognize the contradiction as deliberate.", "output": "Step 1: Sarcasm operates by uttering a statement whose literal meaning is the opposite of the speaker's intended meaning. Step 2: Example: During a torrential downpour, someone says, 'Beautiful day, isn't it?' Step 3: Literal semantic state: The speaker is asserting that the day is beautiful (positive evaluation of weather). Step 4: Intended semantic state: The speaker means the exact opposite — the weather is terrible, and they are complaining ironically. Step 5: To understand this utterance correctly, the LLM must simultaneously represent BOTH meanings in its activation space. Step 6: It must recognize the literal meaning (what the words say) AND the intended meaning (what the speaker actually means). Step 7: Then it must detect the contradiction between these two states and infer that the contradiction is deliberate, signaling sarcasm rather than error or confusion. Step 8: Models without this capability either (a) take the statement literally and fail to detect sarcasm, or (b) classify all contradictions as errors. Training on SARC teaches the model to maintain and compare these dual representations."},
+{"type": "english_reasoning", "instruction": "Why do idioms present a 'semantic opacity' problem for LLMs, and how does varied contextual training solve it? Provide logical steps.", "input": "", "reasoning": "Idioms like 'spill the beans' cannot be understood from 'spill' + 'beans'. Fixed training on one context leads to pattern matching. Varied contexts force the model to abstract the non-literal meaning from diverse syntactic/semantic environments.", "output": "Step 1: Idioms are semantically opaque — their meaning cannot be derived from the literal definitions of their constituent words. For example, 'spill the beans' has nothing to do with spilling or beans; it means 'reveal a secret'. Step 2: If an LLM sees an idiom only in one fixed context (e.g., 'He spilled the beans about the surprise'), it may memorize the pattern 'spill the beans' → 'reveal secret' without understanding the underlying mapping. Step 3: When it encounters a novel syntactic variation ('The beans were spilled by the whistleblower'), it fails because the pattern is disrupted. Step 4: MAGPIE solves this by providing each of 1,756 idioms across multiple, highly varied textual contexts — different syntactic positions, different surrounding vocabulary, different semantic domains. Step 5: By seeing 'spill the beans' in 30+ distinct contexts, the model abstracts the invariant non-literal meaning from the variable surface forms. Step 6: This forces genuine idiom comprehension rather than shallow pattern matching, enabling the model to understand 'the beans were spilled' as readily as 'he spilled the beans.'"},
+{"type": "english_reasoning", "instruction": "Why does the document claim that 'ignorance of profanity leads to a brittle, easily bypassed model' while 'total mastery of it allows for nuanced content moderation'? Provide logical steps.", "input": "", "reasoning": "Ignorant models either block all profanity (easily bypassed with misspellings) or allow all (dangerous). Mastery means understanding severity, context, and intent — enabling gradient responses from blocking to rewriting to allowing based on situation.", "output": "Step 1: An LLM trained without profanity data has no representation of offensive words in its vocabulary or embedding space. Step 2: Such a model cannot recognize profanity when it appears in user inputs — it sees 'f---' or 's--t' as unknown tokens or normal words. Step 3: Malicious users exploit this by using misspelled profanity, homophones (e.g., 'sheet' for 'shit'), or creative variations to bypass content filters. The model is 'brittle' because it fails at the exact edge cases it needs to catch. Step 4: Conversely, a model with total mastery of profanity (via Surge AI and Jigsaw) knows: (a) the exact lexical form of every profanity, (b) its severity score (1-3), (c) its semantic target, and (d) contextual indicators of hostile intent. Step 5: This enables gradient moderation: severity 1 in friendly context → allow; severity 1 in hostile context → warn; severity 3 → block entirely; identity-based hate → rewrite or refuse. Step 6: The model can also distinguish 'I f---ing love this' (positive emphasis) from 'You are a f---ing [slur]' (targeted harassment). Step 7: Thus, mastery enables nuanced, context-aware safety, while ignorance creates easily bypassed filters."},
+{"type": "english_reasoning", "instruction": "Why does the document claim that 'ignorance of profanity leads to a brittle, easily bypassed model' while simultaneously arguing that profanity datasets are essential for world-class English? Resolve this apparent paradox.", "input": "", "reasoning": "There is no paradox: 'world-class English' includes understanding all registers, including profanity. Ignorance means inability to recognize or moderate profanity. Mastery means knowing severity, context, and appropriate usage — enabling nuanced safety, not endorsement of abuse.", "output": "Step 1: The apparent paradox arises from conflating 'knowing profanity' with 'endorsing profanity.' Step 2: A model ignorant of profanity has no representation of offensive words in its vocabulary — it cannot recognize them in user inputs. Step 3: This ignorance creates safety vulnerabilities: malicious users can use profanity to attack others, and the model cannot detect or moderate it. Step 4: Furthermore, the model cannot generate period-authentic dialogue for creative writing (e.g., a Quentin Tarantino script) or understand user intent when profanity is used harmlessly ('This is f---ing amazing!'). Step 5: A model with profanity mastery (via Surge AI and Jigsaw) knows: severity scores (1-3), semantic targets (sexual, scatological, identity-based), and contextual indicators of hostile intent. Step 6: This enables nuanced moderation: the model can block severe targeted harassment (severity 3, identity-based hate), rewrite milder profanity in formal contexts, and allow casual profanity in appropriate roleplay scenarios. Step 7: Thus, mastery enables ethical boundary adherence; ignorance enables neither safety nor authentic communication. World-class English requires understanding ALL registers — including profanity — so the model can navigate them appropriately, not so it can abuse them."},
+{"type": "english_reasoning", "instruction": "Why does the FineWeb-Edu classifier use a regression head on Snowflake-arctic-embed-m embeddings rather than a pure classification model? Provide logical steps.", "input": "", "reasoning": "Regression allows continuous 0–5 scoring rather than binary educational/non-educational labels, enabling threshold-based filtering. The linear regressor is built on frozen embeddings for efficiency.", "output": "Step 1: Educational quality is a spectrum, not a binary label. Some texts are moderately educational (score 3), others highly educational (score 5). Step 2: A binary classifier would collapse this nuance, forcing texts into 'educational' or 'not educational' categories. Step 3: A regression head with a 0–5 output preserves granularity. Step 4: The classifier uses a frozen Snowflake-arctic-embed-m encoder and trains only the regression head for 20 epochs, preserving the embedding space's semantic richness. Step 5: After training, documents with scores ≥3 are retained as FineWeb-Edu. This allows flexible threshold adjustment and yields an 82% binary F1 score when thresholded at 3."},
+{"type": "english_reasoning", "instruction": "Why does Common Corpus include Open Code (283B tokens) when the document focuses on English proficiency? Provide logical steps.", "input": "", "reasoning": "Code data teaches structured reasoning, formal grammar, and unambiguous syntax. Programming languages share formal properties with mathematical and technical English, improving the model's precision in technical domains.", "output": "Step 1: English proficiency at a PhD level includes technical and mathematical English, not just literary or conversational registers. Step 2: Programming languages embody strict formal grammars, unambiguous semantics, and logical control flow — properties that mirror mathematical proofs and technical specifications. Step 3: Training on 283 billion tokens of code across 600+ languages exposes the model to variable naming conventions, comment structures, documentation strings, and error messages. Step 4: This cross-training improves the model's ability to generate and understand highly structured, unambiguous technical English. Step 5: Models trained on code data consistently perform better on reasoning tasks (MMLU, ARC) and mathematical benchmarks, directly supporting the document's goal of graduate-level English proficiency."},
+{"type": "english_reasoning", "instruction": "Why is structured full text (with resolved bibliographic references) more valuable for PhD-level English training than plain abstracts alone? Provide logical steps.", "input": "", "reasoning": "Abstracts summarize but omit methodological details and argument structure. Full text with resolved citations teaches how claims are supported by evidence, how papers build on prior work, and the discourse structure of scientific writing.", "output": "Step 1: Abstracts provide condensed summaries but lack the detailed argumentation, methodology, and evidence chains found in full papers. Step 2: PhD-level English requires understanding how scientific claims are supported — through experimental results, mathematical derivations, or citations to prior work. Step 3: Structured full text with resolved bibliographic references explicitly links each claim to its supporting evidence. Step 4: The model learns discourse patterns: 'As shown in Figure 3', 'consistent with previous findings (Smith et al., 2020)', 'therefore we conclude that...'. Step 5: By observing citation contexts, the model learns which claims require external support and how to integrate references naturally. Step 6: This produces a model that generates citation-dense, evidence-backed scientific prose rather than unsupported assertions."},
+{"type": "english_reasoning", "instruction": "Why does predicting part-of-speech tags before dependency labels improve parsing accuracy? Provide a cognitive reasoning chain.", "input": "", "reasoning": "POS tags provide category-level constraints (e.g., only nouns can be subjects, only verbs can be roots). This reduces the search space for dependency relations and enforces linguistic consistency.", "output": "Step 1: Dependency parsing requires identifying which word is the root and how all other words relate to their syntactic heads. Step 2: Without POS information, the model must consider all possible head-dependent pairs simultaneously. Step 3: POS tags provide category-level constraints: only verbs can be roots, only nouns can be nominal subjects (nsubj), only adjectives can be adjectival modifiers (amod). Step 4: By forcing the model to predict POS tags first, it must commit to word categories before relations. Step 5: These POS predictions reduce the dependency search space: given a verb token, possible dependency relations include root, aux, xcomp; given a noun, possible relations include nsubj, obj, nmod. Step 6: This stepwise decomposition mimics how humans parse sentences: first identify word classes, then assign grammatical relations. Step 7: The result is higher accuracy and elimination of hallucinated relations that violate POS constraints."},
+{"type": "english_reasoning", "instruction": "Why does training on out-of-distribution SFT data lead to diminishing returns or performance degradation? Provide logical steps.", "input": "", "reasoning": "Out-of-distribution data forces the model to override its learned representations, causing catastrophic forgetting or degraded robustness. GRAPE avoids this by selecting in-distribution responses.", "output": "Step 1: A pretrained LLM has learned a specific probability distribution over tokens from vast pretraining on diverse text. Step 2: Standard SFT data often comes from other LLMs or humans, producing responses that may differ stylistically, structurally, or factually from what the target model would naturally generate. Step 3: When the model fine-tunes on such out-of-distribution data, it must adjust its weights to assign higher probability to response patterns that are foreign to its learned distribution. Step 4: This adjustment can overwrite or degrade the original capabilities (catastrophic forgetting) and may produce brittle performance on tasks not covered by the SFT data. Step 5: GRAPE selects responses that the target model already assigns high probability to — these responses are 'in distribution' by definition. Step 6: Fine-tuning on in-distribution data requires minimal weight adjustment, preserving original capabilities while adding instruction-following. Step 7: Hence, GRAPE achieves superior results with less data, avoiding the diminishing returns of out-of-distribution SFT."},
+{"type": "english_reasoning", "instruction": "Why does The Pile exclude low-quality web data in favor of curated sources, unlike FineWeb which includes filtered web text? Compare the two approaches.", "input": "", "reasoning": "The Pile selects only high-quality domains (academic, legal, code). FineWeb includes all web text but applies educational filtering. The Pile's approach avoids web noise entirely; FineWeb's approach scales but requires sophisticated filtering.", "output": "Step 1: The Pile and FineWeb represent two different philosophies for pretraining data curation. Step 2: The Pile selects only specific high-quality domains (PubMed, arXiv, GitHub, USPTO, FreeLaw) and explicitly excludes general web text. This guarantees quality but limits scale and diversity of writing styles. Step 3: FineWeb (and its FineWeb-Edu subset) includes web text from Common Crawl but applies filtering (educational scoring, heuristics, deduplication). This allows massive scale (15 trillion tokens raw, 1.3 trillion filtered) but risks residual web noise. Step 4: The Pile's approach is safer for domain-specific tasks (legal, medical, code) but may miss the linguistic diversity of web-written English (blogs, forums, news). Step 5: FineWeb's approach captures broader registers but requires more sophisticated filtering. Step 6: Optimal pretraining may combine both: The Pile for high-precision domains, FineWeb-Edu for general educational text, and Common Corpus for ethically licensed diversity."},
+{"type": "english_reasoning", "instruction": "Why is sample-wise rule generation (ProX) more effective than fixed heuristics for data cleaning at scale? Provide logical steps.", "input": "", "reasoning": "Fixed heuristics apply same rules to all documents, missing document-specific noise patterns. ProX generates custom cleaning programs per document, adapting to unique formatting anomalies.", "output": "Step 1: Web-sourced text contains diverse, document-specific noise patterns: some have HTML tables, others have LaTeX math, others have JSON snippets, others have multilingual sections. Step 2: Fixed heuristics (e.g., 'remove all HTML tags') apply the same transformation to every document, either missing noise (if the heuristic is too narrow) or over-cleaning (if too aggressive). Step 3: ProX treats each document as a unique case, generating a custom cleaning program for that document only. Step 4: The 0.3B LM examines the document content and writes specific rules: for a document with markdown tables, it generates table-flattening code; for a document with JSON, it generates JSON extraction code; for a document with LaTeX, it generates LaTeX-to-plaintext conversion. Step 5: Each program is executed, and the cleaned document is produced. Step 6: This sample-wise adaptation handles the long tail of formatting anomalies that fixed heuristics miss, producing a higher-quality corpus with fewer artifacts."}

english_summary.jsonl

@@ -0,0 +1,19 @@
+{"type": "english_summarize", "instruction": "Summarize the main claim made in the first paragraph of the document about achieving world-class English proficiency in LLMs.", "input": "", "reasoning": "The paragraph argues that a paradigm shift from raw data volume to meticulous data curation is necessary for graduate-level reasoning and native-like grasp of English.", "output": "To build an LLM with world-class English proficiency (graduate-level reasoning, grammatical precision, deep contemplation, and colloquial pragmatics), researchers must abandon the brute-force method of unfiltered web data. Instead, a paradigm shift toward meticulously stratified data curation is required to overcome the functional brittleness of statistically fluent models."},
+{"type": "english_summarize", "instruction": "Summarize the empirical benefits of pre-training on FineWeb-Edu as reported in the document.", "input": "", "reasoning": "The document states a ~12% relative gain on MMLU (33% to 37% accuracy) and a 24% gain on ARC (46% to 57%).", "output": "Pre-training on FineWeb-Edu yields significant performance improvements: a 12% relative gain on the Massive Multitask Language Understanding (MMLU) benchmark, raising accuracy from 33% to 37%, and a 24% gain on the AI2 Reasoning Challenge (ARC), raising scores from 46% to 57%. These gains demonstrate that educational filtering directly enhances academic reasoning and factual consistency."},
+{"type": "english_summarize", "instruction": "Summarize the domain-specific subsets of the Common Corpus and their linguistic benefits.", "input": "", "reasoning": "Open Culture (885.9B tokens): classic literature, historical newspapers → teaches archaic vocabulary and classical syntax. Open Government (406.6B tokens): financial and legal documents → teaches formal, precise professional register.", "output": "The Common Corpus includes two key English subsets: (1) Open Culture with 885.9 billion tokens from classic literature, historical newspapers, and cultural heritage texts — preventing temporal myopia by teaching archaic vocabulary and classical sentence structures. (2) Open Government with 406.6 billion tokens from financial documents (Finance Commons) and legal case law (Legal Commons from Free Law Project, EUR-Lex, USPTO) — enabling the model to adopt a highly formal, precise, professional register for complex regulatory or economic subjects."},
+{"type": "english_summarize", "instruction": "Summarize the key findings about the Lang-8 dataset and its role in grammatical error detection.", "input": "", "reasoning": "Lang-8 has 2.3M rows of learner errors corrected by native speakers. Fine-tuning on cleaned versions achieves 0.91 F1 score and 90.53% accuracy for error detection. Native vs non-native error patterns differ.", "output": "The Lang-8 dataset contains over 2.3 million rows of language learner errors manually corrected by native speakers. Fine-tuning transformer models on cleaned iterations achieves an F1 score of 0.91 and 90.53% accuracy for Grammatical Error Detection. Critically, native and non-native speakers make different error types: native speakers struggle with punctuation and spelling, while non-native speakers struggle with determiners and articles. Training on Lang-8 ensures robustness against all error types."},
+{"type": "english_summarize", "instruction": "Summarize what bridging anaphora is and how the GUMBridge corpus trains LLMs to handle it.", "input": "", "reasoning": "Bridging anaphora: understanding a new entity via its relation to a previous non-identical entity (e.g., 'house' → 'door'). GUMBridge provides granular annotations across 24 genres to teach deep contextual memory and coreference resolution.", "output": "Bridging anaphora is a linguistic phenomenon where a newly introduced entity is understood solely through its relation to a previously mentioned, non-identical entity (e.g., 'There is a house. The door is red' — 'the door' is inferable as belonging to 'the house'). The GUMBridge corpus, a subset of the Georgetown University Multilayer corpus, spans 24 diverse English genres with granular annotations for bridging anaphora. Training on GUMBridge forces the LLM to develop deep contextual memory and complex coreference resolution skills, marking the dividing line between disjointed sentence generation and cohesive, logically flowing paragraphs."},
+{"type": "english_summarize", "instruction": "Summarize the GPQA and MMLU-Pro benchmarks and how they evaluate graduate-level reasoning.", "input": "", "reasoning": "GPQA contains graduate-level questions in physics, chemistry, biology designed to be unsolvable by search engines. MMLU-Pro expands from 4 to 10 answer options across 12,000 questions in 14 fields, requiring extreme confidence and precision.", "output": "GPQA (Graduate-Level Google-Proof Q&A) contains highly complex questions in physics, chemistry, and biology requiring deep subject-matter knowledge, calculation, and concept synthesis. These questions are explicitly designed to be unsolvable by simple search engines, forcing the model to rely on internalized academic logic. MMLU-Pro represents a significant escalation from the original MMLU benchmark: it includes 12,000 high-quality academic exam questions spanning 14 major fields, with answer options expanded from 4 to 10. This expanded distractor set eliminates random guessing or shallow heuristic matching, requiring extreme confidence and precision in knowledge retrieval."},
+{"type": "english_summarize", "instruction": "Summarize the GRAPE framework and its efficiency benefits for instruction tuning.", "input": "", "reasoning": "GRAPE aligns SFT data with the model's pre-trained distribution by selecting responses that match the target model's probability distribution. It achieves up to 13.8% gains and requires 1/3 the data and half the epochs to exceed baselines trained on 4.5x more data.", "output": "The GRAPE framework revolutionizes Supervised Fine-Tuning (SFT) by aligning instructional data with the model's inherent pre-trained probability distribution. For each instruction, GRAPE gathers responses from multiple sources (human annotators or larger oracle models) and selects the specific response that most closely matches the target model's pre-trained distribution. Standard SFT then runs only on these probabilistically aligned responses. In controlled experiments on Ultralnteract and OpenHermes-2.5, GRAPE-selected data yielded absolute gains up to 13.8% across benchmarks. Remarkably, using GRAPE to subsample post-training data, a smaller model (LLaMA-1-8B) required only one-third of the data and half the training epochs to exceed baselines trained on 4.5 times more data."},
+{"type": "english_summarize", "instruction": "Summarize the LiteraryTaste dataset and how it captures nuanced human preferences for writing style.", "input": "", "reasoning": "LiteraryTaste contains 100 pairs of short creative writing texts annotated by 60 individuals revealing revealed vs stated reading preferences. Fine-tuning on it achieves >75% accuracy in modeling personal preferences, allowing dynamic register shifting.", "output": "The LiteraryTaste dataset captures nuanced human preferences for creative writing style. It contains 100 pairs of short creative writing texts, annotated by 60 individuals who detailed their revealed preferences (what they actually enjoyed reading) versus stated preferences (what they claimed to like). Fine-tuning a transformer encoder on this dataset achieves over 75% accuracy in modeling revealed personal preferences. This allows the LLM to adapt its prose dynamically — shifting seamlessly from the ornate, polysyllabic elegance of Victorian literature to the punchy, minimalist clarity of modern journalism — based entirely on the user's implicit needs and preferences."},
+{"type": "english_summarize", "instruction": "Summarize the PodSarc dataset and how it complements SARC for sarcasm detection.", "input": "", "reasoning": "PodSarc provides large-scale sarcastic speech from podcasts with average 31.18 words over 9.61 seconds per segment. It captures spoken English nuances: conversational shifts, rhetorical questions, deadpan delivery — things absent from written text.", "output": "The PodSarc dataset offers a large-scale collection of sarcastic speech derived from podcasts. With an average of 31.18 words spanning 9.61 seconds per segment, this dataset provides lengthy, dialogue-heavy contexts necessary for detecting conversational shifts, rhetorical questions, and deadpan delivery. It complements SARC by capturing the nuances of spoken English translated into text — features like prosody, timing, and intonation patterns that are implicit in speech but absent from written Reddit comments. Together, these datasets ensure the LLM can detect sarcasm whether it appears in informal online writing or transcribed conversation, a critical capability for true native-level pragmatic understanding."},
+{"type": "english_summarize", "instruction": "Summarize how the MAGPIE and IDEM datasets train LLMs to understand idiomatic expressions.", "input": "", "reasoning": "MAGPIE provides 56,000+ instances of 1,756 unique English idioms in varied contexts. IDEM attaches emotion labels to nearly 10,000 idiom-containing sentences, teaching the affective weight of figurative language.", "output": "MAGPIE addresses the challenge of semantic opacity in idioms — where meaning cannot be deduced from literal word definitions — by providing over 56,000 instances of 1,756 unique English idioms presented within short, highly varied textual contexts. This exposes the model to idioms in diverse syntactic and semantic environments, preventing memorization of fixed patterns. Complementing this, the IDEM dataset attaches specific emotion labels to nearly 10,000 idiom-containing sentences, teaching the model the affective weight and emotional resonance of figurative language — for example, that 'kicked the bucket' carries different pragmatic force than 'passed away' despite similar literal reference. Together, these datasets ensure the LLM understands both the non-literal meaning and emotional register of idiomatic expressions."},
+{"type": "english_summarize", "instruction": "Summarize the Jigsaw Toxic Comment Classification dataset and its role in teaching ethical boundary adherence.", "input": "", "reasoning": "Jigsaw contains 160,000 human-annotated Wikipedia talk page comments categorized into sub-labels: severe toxicity, insults, threats, identity-based hate. It teaches how benign words can be weaponized and how profanity can be harmless in colloquial contexts.", "output": "The Jigsaw Toxic Comment Classification dataset contains 160,000 human-annotated Wikipedia talk page comments, categorized into distinct sub-labels including severe toxicity, insults, threats, and identity-based hate. This dataset serves as the primary mechanism for teaching the model nuanced content moderation: how seemingly benign words can be weaponized into abuse through context (e.g., 'interesting' said sarcastically to dismiss someone's identity), and conversely, how profanity can be used harmlessly for emphasis in colloquial contexts (e.g., 'That's f---ing awesome!' as positive exclamation). Training on Jigsaw ensures the LLM possesses ethical discernment and pragmatic awareness — knowing exactly when offensive language is unacceptable (harassment, hate speech) versus when it might be appropriate (creative writing, specific roleplay scenarios, casual conversation among consenting adults)."},
+{"type": "english_summarize", "instruction": "Summarize the document's core argument about why indiscriminate scaling fails and what succeeds instead.", "input": "", "reasoning": "Indiscriminate scaling (more parameters, more raw web data) yields diminishing returns because noisy data teaches surface patterns, not deep understanding. Success requires meticulous, multi-stage data curation with specialized datasets for syntax, reasoning, style, and pragmatics.", "output": "The document's core argument is that achieving world-class English proficiency in LLMs cannot be achieved by indiscriminately expanding context windows or increasing parameter counts over raw, noisy web data. Such brute-force approaches yield statistically fluent but functionally brittle models that fail at sub-surface complexities like sarcasm, bridging anaphora, or graduate-level reasoning. Instead, success requires a meticulous, multi-stage data curation paradigm: (1) educationally filtered pre-training for structured exposition, (2) syntactic datasets for grammatical perfection and coreference resolution, (3) Chain-of-Thought and reasoning datasets for step-by-step logic, (4) literary and philosophical corpora for elegant prose, and (5) pragmatic datasets for sarcasm, humor, idioms, and profanity. Only this carefully sequenced symphony of specialized datasets produces a model that is genuinely 'PhD-smart,' deeply thoughtful, and capable of native-like communication."},
+{"type": "english_summarize", "instruction": "Summarize the benchmarking performance of FineWeb-Edu on MMLU and other reasoning tasks.", "input": "", "reasoning": "FineWeb-Edu improves MMLU from 33% to approximately 37% and shows substantial gains on ARC and OpenBookQA.", "output": "FineWeb-Edu demonstrates substantial empirical gains on knowledge- and reasoning-intensive benchmarks. On MMLU, models pretrained on FineWeb-Edu improve from 33% to approximately 37% accuracy. It also yields significant improvements on the AI2 Reasoning Challenge (ARC) and OpenBookQA, making it one of the most effective open educational web datasets for LLM pretraining."},
+{"type": "english_summarize", "instruction": "Summarize the licensing and ethical compliance features of Common Corpus.", "input": "", "reasoning": "All components are either public domain or under permissive licenses (CC-By, CC0, MIT, Apache, BSD). Extensive provenance documentation ensures legal and ethical compliance for commercial AI deployment.", "output": "Common Corpus is assembled exclusively from uncopyrighted or permissively licensed sources. All components are either public domain or distributed under licenses such as CC-By, CC0, MIT, Apache, or BSD. The dataset includes exhaustive documentation of provenance, licensing, and curation procedures, ensuring compliance with data security regulations and intellectual property laws. This design makes Common Corpus suitable for both scientific research and commercial AI deployment under emerging AI legislation."},
+{"type": "english_summarize", "instruction": "Summarize the relationship between S2ORC, the Semantic Scholar Academic Graph (S2AG), and the overall Ai2 open data ecosystem.", "input": "", "reasoning": "S2AG provides 200M paper titles/abstracts/citations as the underlying graph. S2ORC adds structured full text for a subset. Together they form the largest open scientific literature graph.", "output": "The Semantic Scholar Academic Graph (S2AG) is a large collection of over 200 million paper titles, abstracts, citations, and other metadata from open access papers, forming the largest open scientific literature graph. S2ORC builds on S2AG by adding structured full text for 8.1 million open access papers, with annotations for citations, figures, and tables. Together, these resources provide a comprehensive, machine-readable representation of the scientific literature, enabling tasks such as citation analysis, knowledge graph construction, and scientific language model pretraining."},
+{"type": "english_summarize", "instruction": "Summarize the step-by-step instruction strategy for improving LLM dependency parsing accuracy using CoNLL-U format.", "input": "", "reasoning": "Recent research shows that requiring LLMs to predict universal part-of-speech tags before syntactic heads and dependency labels improves accuracy across 17 languages without hallucination.", "output": "A 2025 study proposes a step-by-step instruction strategy for LLM-based dependency parsing: first, predict universal part-of-speech (UPOS) tags for each token; second, predict syntactic heads and dependency labels using the UPOS tags as intermediate reasoning steps. This method uses a simplified CoNLL-U-like output format and achieves state-of-the-art accuracy on Universal Dependencies datasets across 17 languages without hallucination or data contamination. The approach highlights the effectiveness of explicit reasoning steps in LLM-based parsing, offering a scalable alternative to bracket-based methods."},
+{"type": "english_summarize", "instruction": "Summarize the empirical performance gains achieved by GRAPE on Llama3.1-8B and other models.", "input": "", "reasoning": "GRAPE achieves absolute gains up to 13.8% across benchmarks, outperforms models trained on 4.5x more data by 6.1%, and enables 1/3 data and half epochs to surpass baseline by 3.5%.", "output": "GRAPE significantly outperforms strong baselines across multiple models (Llama3.1-8B, Mistral-7B, Qwen2.5-7B). It achieves absolute gains up to 13.8% averaging across benchmarks compared to distilling from the strongest model, and maximum 17.3% improvements over training on 3 times more data. In realistic settings using post-training data for Tulu3 and Olmo-2, GRAPE outperforms baselines with 4.5 times the data by 6.1% and outperforms state-of-the-art data selection approaches by 3.9% on average. Remarkably, using only 1/3 of the data and half the number of epochs, GRAPE allows Llama3.1-8B to surpass Tulu3-SFT by 3.5%."},
+{"type": "english_summarize", "instruction": "Summarize the Dolma dataset and its role as a 3-trillion-token open corpus for LLM pretraining.", "input": "", "reasoning": "Dolma is a 3-trillion-token open corpus from Common Crawl, academic publications, code, and social media. It emphasizes transparency with detailed documentation of curation decisions.", "output": "Dolma is an open corpus of three trillion tokens for language model pretraining research, released by the Allen Institute for AI. It aggregates data from Common Crawl, academic publications (via S2ORC), code repositories (GitHub), and social media (Reddit, Wikipedia). The corpus emphasizes transparency, providing detailed documentation of all curation decisions including filtering criteria, deduplication methods, and tokenization strategies. This enables reproducible pretraining experiments and direct comparison of data curation techniques."},
+{"type": "english_summarize", "instruction": "Summarize the RedPajama-V2 dataset and its quality filtering approaches for open LLM pretraining.", "input": "", "reasoning": "RedPajama-V2 is an open dataset of 30 trillion tokens from Common Crawl, using multiple quality filters: exact deduplication, heuristics, classifier-based, and MinHash.", "output": "RedPajama-V2 is an open dataset for LLM pretraining, containing 30 trillion raw tokens from Common Crawl (July 2020–October 2023). It applies multiple quality filtering approaches: exact substring deduplication removing duplicated passages; heuristic quality scores based on document statistics; classifier-based filtering using fastText and a linear model; and MinHash deduplication with 5-grams. Processed subsets are available at different quality thresholds, enabling experiments on the trade-off between data quantity and quality. The dataset includes 38 billion quality-filtered tokens (high-quality subset) and 112 billion deduplicated tokens (medium-quality)."}

instruction_following.jsonl

@@ -0,0 +1,19 @@
+{"type": "english_instruction", "instruction": "Write a short paragraph describing the required paradigm shift in data curation as outlined in the introduction. Use formal academic tone.", "input": "", "reasoning": "The shift is from monolithic web scraping to a multi-stage pipeline: foundational pre-training for syntax, instruction tuning for reasoning, stylistic alignment for prose, and pragmatic disambiguation for sarcasm/idioms.", "output": "The necessary paradigm shift entails moving beyond monolithic web scraping toward a meticulously stratified data pipeline. This architecture comprises progressive stages: foundational pre-training for structural syntax, targeted instruction tuning for academic rigor and deep reasoning, stylistic alignment for world-class literary prose, and pragmatic disambiguation to handle sarcasm, humor, idioms, and profanity. Only through such a curated curriculum can an LLM transcend stochastic parroting and emerge as a nuanced, deeply thoughtful linguistic entity."},
+{"type": "english_instruction", "instruction": "Write a technical note describing the curation pipeline of FineWeb-Edu, including the model used for annotation and the scoring mechanism.", "input": "", "reasoning": "Pipeline: Common Crawl → FineWeb (15T tokens) → sample 460k texts → annotate with Llama-3-70B-Instruct (additive score 0-5) → train linear regressor on Snowflake embeddings → predict scores for all → filter score ≥3 → FineWeb-Edu (5.4T tokens).", "output": "Curation pipeline for FineWeb-Edu: 1) Start from FineWeb, a 15-trillion-token corpus derived from 96 Common Crawl snapshots. 2) Sample 460,000 candidate texts. 3) Use Llama-3-70B-Instruct to annotate each text with an additive educational score (0-5). 4) Train a linear regressor on Snowflake-arctic-embed-m embeddings to predict scores for the entire FineWeb corpus. 5) Apply a threshold of 3 or higher. 6) Retain 5.4 trillion tokens as FineWeb-Edu. This automated, scalable filtering prioritizes academic and structured content over noisy web text."},
+{"type": "english_instruction", "instruction": "Write a brief summary of the structural diversity datasets mentioned after Common Corpus (The Pile, GneissWeb, ProX framework). Include their sizes and key features.", "input": "", "reasoning": "The Pile: 825 GB, 22 domains (arXiv, PubMed, GitHub). GneissWeb: ~10T tokens, exact substring deduplication + quality filters. ProX: 0.3B-parameter LM that writes cleaning programs per record.", "output": "The Pile (825 GB) combines 22 diverse high-quality sources including academic publications (arXiv, PubMed Central), GitHub repositories, and medical text, improving cross-disciplinary vocabulary. GneissWeb (~10 trillion tokens) uses exact substring deduplication and an ensemble of quality filters, outperforming FineWeb-V1.1.0 by 2.73 percentage points across 11 benchmarks. The ProX framework (Programming Every Example) uses a lightweight 0.3B-parameter language model to write and execute targeted cleaning programs (e.g., HTML stripping, string normalization) for each record in datasets like C4, RedPajama-V2, and DCLM, ensuring a pristine foundational corpus."},
+{"type": "english_instruction", "instruction": "Write a technical description of the C4_200M synthetic dataset and its purpose in grammatical error correction training.", "input": "", "reasoning": "C4_200M contains 200 million sentence pairs with synthetically induced grammatical errors. It is massive and complements real learner corpora by providing controlled error types at scale.", "output": "The C4_200M synthetic dataset comprises 200 million sentence pairs where grammatical errors have been artificially introduced into clean text. Each pair consists of an error-containing sentence and its corrected version. The synthetic generation allows controlled sampling of specific error types (e.g., tense mismatches, subject-verb disagreement, punctuation errors). This dataset serves as a massive complement to real learner corpora like Lang-8, enabling the LLM to encounter a diverse, balanced distribution of grammatical mistakes. Because native and non-native error patterns differ, C4_200M ensures the model is robust against all forms of input degradation, making it capable of correcting errors from any user population."},
+{"type": "english_instruction", "instruction": "Write a technical explanation of the SCAR framework and how it selects training data for stylistic consistency.", "input": "", "reasoning": "SCAR (Style Consistency-Aware Response Ranking) ranks instruction-response pairs by linguistic form and instructional surprisal, selecting the most style-consistent examples. Sometimes only 0.7% of data is needed to match full-dataset performance.", "output": "The SCAR (Style Consistency-Aware Response Ranking) framework automates data selection for stylistic tuning. It operates on two key elements: (1) linguistic form — the surface features of text (punctuation, sentence length, register markers); and (2) instructional surprisal — how unexpected the response is given the instruction. SCAR ranks all instruction-response pairs in a training set by their stylistic consistency, then selects the most consistent examples. In practice, fine-tuning LLMs on as little as 0.7% of the full dataset — the highest-ranked examples — matches or surpasses the performance of models trained on entire, uncurated datasets. This efficiency comes from eliminating stylistically inconsistent or noisy examples that would otherwise degrade the model's register adherence."},
+{"type": "english_instruction", "instruction": "Write a technical description of the FACT and RACE evaluation frameworks introduced in DeepResearch Bench.", "input": "", "reasoning": "DeepResearch Bench provides 100 PhD-level research tasks across 22 fields. RACE (Reference-based Adaptive Criteria-driven Evaluation) assesses report quality. FACT (Framework for Factual Abundance and Citation Trustworthiness) measures citation count and accuracy.", "output": "DeepResearch Bench introduces two novel evaluation frameworks for assessing deep research agents. RACE (Reference-based Adaptive Criteria-driven Evaluation) evaluates the quality of generated research reports by adaptively applying criteria based on reference answers — it measures coherence, completeness, and methodological soundness. FACT (Framework for Factual Abundance and Citation Trustworthiness) assesses the model's citation practices through two metrics: effective citation count (how many claims are properly supported) and overall citation accuracy (whether citations actually support the claims they accompany). By fine-tuning LLMs on data scoring highly on FACT, the model learns to automatically embed context, origin, and mechanism into its answers, mimicking the rigorous citation standards of peer-reviewed literature rather than generating unsupported assertions."},
+{"type": "english_instruction", "instruction": "Write a technical description of the LongPage dataset and how it enables hierarchical narrative planning over extended contexts.", "input": "", "reasoning": "LongPage provides 300 full-length books (40k-600k tokens each) with hierarchical reasoning traces: character archetypes, world rules, scene breakdowns, thematic coherence. These act as 'Chain of Thought for creative writing' over massive context windows.", "output": "The LongPage dataset addresses long-horizon reasoning for creative writing by providing 300 full-length books, each ranging from 40,000 to over 600,000 tokens. Crucially, each book is accompanied by hierarchical reasoning traces — structured annotations that map narrative logic across the entire text. These traces include character archetypes (personality consistency), world rules (setting constraints), scene breakdowns (event sequencing), and thematic coherence (motif tracking). Functionally, these traces act as 'Chain of Thought for creative writing,' explicitly showing the model how to plan and maintain logical consistency over massive context windows. Training on LongPage ensures the model does not lose its train of thought when deep-diving into complex, multi-page English responses — whether writing a novel, a legal brief, or a philosophical treatise."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Wiki Neutrality Corpus (WNC) and its role in teaching objective, academic prose.", "input": "", "reasoning": "WNC is used for neutralizing subjectivity bias. It teaches the model to identify emotionally loaded terms (e.g., 'claimed' vs 'stated', 'controversial' vs 'notable') and replace them with objective, academic equivalents.", "output": "The Wiki Neutrality Corpus (WNC) is a Text Style Transfer dataset specifically designed for neutralizing subjectivity bias in writing. It consists of pairs of Wikipedia sentences: one version containing biased or emotionally loaded language, and a neutralized version that adheres to Wikipedia's Neutral Point of View policy. Training on WNC teaches the LLM to identify subjective language patterns — loaded adjectives ('horrific', 'brilliant'), framing verbs ('claimed' vs 'stated', 'admitted' vs 'explained'), and evaluative hedges ('controversially', 'fortunately'). The model learns to replace these with objective, academic equivalents: 'claimed' becomes 'stated', 'horrific event' becomes 'event', 'fortunately' is removed entirely. This capability is essential for generating scholarly prose that prioritizes factual reporting over rhetorical persuasion, a hallmark of world-class academic English."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Ironic Corpus and how it distinguishes sarcasm from situational irony.", "input": "", "reasoning": "The Ironic Corpus provides human-annotated labels distinguishing situational irony (events contradicting expectations) from sarcasm (verbal statements contradicting intended meaning). This prevents models from conflating all non-literal text with sarcasm.", "output": "The Ironic Corpus is a human-annotated dataset designed to distinguish between closely related but distinct rhetorical devices. It provides fine-grained labels separating situational irony (an event or outcome that contradicts expected or intended results — e.g., a fire station burning down) from verbal sarcasm (a statement whose literal meaning opposes the speaker's intended meaning — e.g., 'Great job' after a failure). This distinction is crucial because both phenomena involve contradiction, but they operate at different levels: irony at the level of events, sarcasm at the level of utterance meaning. Without explicit training on this distinction, LLMs conflate all non-literal, contradictory-sounding text with sarcasm, leading to false positives. The Ironic Corpus provides the annotation granularity needed for the model to correctly classify each device based on whether the contradiction is situational or verbal, enabling precise pragmatic understanding across diverse rhetorical forms."},
+{"type": "english_instruction", "instruction": "Write a technical description of the SlangLLM framework and the OpenSubtitles-Slang dataset for training contemporary colloquial understanding.", "input": "", "reasoning": "OpenSubtitles-Slang provides slang in messy real-world conversational contexts from movie/TV subtitles. SlangLLM is a framework for dynamic detection and contextual filtering of slang, mapping rapidly evolving internet vernacular.", "output": "The SlangLLM framework, trained on the OpenSubtitles-Slang dataset, addresses the challenge of rapidly evolving contemporary slang in natural language processing. OpenSubtitles-Slang extracts slang usage from messy, real-world conversational contexts within movie and television subtitles — capturing how slang actually appears in natural dialogue (with interruptions, false starts, overlapping speech, and register mixing). This contrasts with curated or dictionary-based slang datasets that present sanitized, context-free definitions. SlangLLM then implements dynamic detection (identifying which words or phrases function as slang in a given context) and contextual filtering (determining when slang is appropriate to generate vs. when formal language is required). This training ensures the model understands the fluidity of modern colloquialisms — including internet-born terms, regional expressions, and generational slang — preventing it from sounding overly robotic, rigid, or anachronistic when users prompt it to adopt a casual, friendly persona."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Pragmatic Category Table from the document, summarizing all six pragmatic dimensions and their datasets.", "input": "", "reasoning": "Table covers Sarcasm/Irony (SARC, PodSarc), Humor/Satire (Deceptive Humor Dataset), Idioms/Slang (MAGPIE, SlangLLM), Profanity/Toxicity (Surge AI, Jigsaw), each with specific annotation structures and proficiency enhancements.", "output": "The Pragmatic Category Table from the document organizes six dimensions of non-literal language mastery for LLMs. (1) Sarcasm/Irony: SARC (1.3M Reddit comments) and PodSarc (9.6-second podcast segments) teach detection of opposite intended meaning via extended conversational context. (2) Humor/Satire: Deceptive Humor Dataset provides Satire Levels 1-3 across five archetypes (Dark Humor, Irony, Social Commentary, Wordplay, Absurdity), enabling generation and comprehension of subtle wordplay, dark humor, and social commentary. (3) Idioms: MAGPIE provides 56,000+ instances of 1,756 unique idioms across varied contexts, preventing literal misinterpretation of semantic opacity. (4) Slang: SlangLLM with OpenSubtitles-Slang maps contemporary internet colloquialisms, ensuring fluid casual interaction. (5) Profanity: Surge AI catalogs 1,600+ profanities with severity scale 1-3 and semantic target categorization. (6) Toxicity: Jigsaw provides 160,000 labeled comments with sub-labels (severe toxicity, insults, threats, identity-based hate). Together, these six dimensions enable an LLM to understand language as it is actually used — with all its irony, humor, idiom, slang, and emotional intensity — while maintaining ethical boundaries."},
+{"type": "english_instruction", "instruction": "Write a final synthesis paragraph describing the ultimate vision of an LLM trained according to this document's paradigm.", "input": "", "reasoning": "The vision is a model that transcends 'stochastic parrot' limitations to become an 'elite, deeply thoughtful, highly nuanced linguistic entity' — capable of graduate-level reasoning, flawless grammar, philosophical contemplation, native-like pragmatics, and ethical discernment across all registers of English.", "output": "The ultimate vision of an LLM trained according to this document's paradigm is a model that transcends the limitations of a stochastic parrot — a system that does not merely predict plausible next tokens but genuinely understands English at a PhD level. This model would demonstrate: graduate-level reasoning on GPQA and MMLU-Pro without relying on shallow heuristics; flawless grammatical precision including mastery of bridging anaphora and complex multi-clause dependencies; deep philosophical contemplation through Socratic dialogue and ethical reasoning; native-like pragmatic competence including accurate sarcasm detection, context-appropriate humor generation, idiom comprehension, and fluid slang usage; and ethical discernment that distinguishes casual banter from targeted harassment. It would shift seamlessly from the ornate elegance of Victorian prose to the minimalist clarity of technical documentation, from the formal register of legal argumentation to the casual warmth of friendly conversation. In short, it would emerge not as a brittle text generator but as an elite, deeply thoughtful, highly nuanced linguistic entity — a model that truly understands English better than the average human and communicates with world-class proficiency, wisdom, and wit."},
+{"type": "english_instruction", "instruction": "Write a technical note describing the classifier training details for FineWeb-Edu, including dataset size, epochs, and F1 scores.", "input": "", "reasoning": "Classifier trained on 450k Llama3 annotations, 20 epochs, learning rate 3e-4, embedding layers frozen. Achieves 82% binary F1 at threshold 3.", "output": "The FineWeb-Edu classifier is trained on 450,000 samples annotated by Llama-3-70B-Instruct with educational scores 0–5. A classification head with single regression output is added to Snowflake-arctic-embed-m. Training runs for 20 epochs with learning rate 3e-4 while embedding and encoder layers remain frozen. The model achieves 82% F1 score when converted to a binary classifier using threshold 3. Weighted average F1 across all score levels is 0.71, with individual class performance varying due to annotation noise."},
+{"type": "english_instruction", "instruction": "Write a technical description of the language coverage in Common Corpus beyond English.", "input": "", "reasoning": "Common Corpus includes major European languages plus low-resource languages (Arabic, Bengali, Latin, Persian, Russian, Sanskrit, Urdu). Open Science is 85% English; Open Culture spans 13+ languages.", "output": "Common Corpus provides multilingual coverage ranging from high-resource European languages to low-resource languages rarely represented in pretraining datasets. Open Culture spans at least 13 major languages (French, English, German, Spanish, Portuguese, Italian, Dutch, Luxembourgish, Danish, Swedish, Serbian, Czech, Greek) plus significant representation in Arabic, Bengali, Latin, Persian, Russian, Sanskrit, and Urdu. Open Science focuses primarily on English (approximately 85%) with substantial French, Spanish, and German content. Open Semantic provides Wikidata natural language renditions in 300 languages. This breadth enables cross-lingual transfer learning and supports the model's understanding of English through comparative linguistic structures."},
+{"type": "english_instruction", "instruction": "Write a technical note on the inline annotation features of S2ORC full text (citations, figures, tables).", "input": "", "reasoning": "Full text annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to corresponding paper objects.", "output": "S2ORC's structured full text includes automatically-detected inline annotations for three key elements. Citation mentions: any reference to another paper (e.g., '(Smith, 2020)') is detected and linked to the resolved bibliographic entry in S2AG. Figure mentions: references to figures within the paper (e.g., 'as shown in Figure 3') are annotated and linked to the figure's location. Table mentions: references to data tables are similarly annotated and linked. These annotations enable the model to learn the discourse functions of citations (supporting claims), figures (visual evidence), and tables (data presentation), critical for generating authentic scientific writing."},
+{"type": "english_instruction", "instruction": "Write a technical description of how bridging anaphora is encoded in CoNLL-U's MISC field using CorefUD standards.", "input": "", "reasoning": "Bridging-specific information is captured by attribute-value pairs in the MISC column, following CorefUD 1.3 conventions for linking anaphors to their implicit antecedents.", "output": "In the CorefUD (Coreference in Universal Dependencies) framework, bridging anaphora annotations are stored as attribute-value pairs in the MISC column of CoNLL-U files. For a bridging anaphor like 'the door' in 'There is a house. The door is red', the annotation includes a bridging link to the previously mentioned 'house' using attributes such as 'Bridge=house' or 'Referent=house'. This encoding allows explicit representation of the inferential relationship between entities that are not coreferential (the door is not the same as the house) but are semantically linked. Multiple bridging relations can be encoded per token, enabling training of LLMs to resolve implicit references across sentence boundaries."},
+{"type": "english_instruction", "instruction": "Write a technical description of GRAPE's response selection process for a single instruction, contrasting it with standard SFT data collection.", "input": "", "reasoning": "Standard SFT collects responses from external LLMs without distribution matching. GRAPE computes target model's perplexity for each candidate response and selects the lowest-perplexity (highest-probability) one.", "output": "In standard SFT data collection, an instruction is paired with a response sampled from a strong external LLM (e.g., GPT-4) or human annotators, regardless of whether that response matches the target model's distribution. In GRAPE, for each instruction, the system first gathers multiple candidate responses from various sources (multiple LLMs, humans, or both). For each candidate, the target model computes the log probability (or perplexity) of the response given the instruction. The candidate with the highest probability (lowest perplexity) is selected — this is the response the target model 'most wants to generate' based on its pretrained distribution. The selected instruction-response pair is then added to the SFT dataset. This ensures every training example is distributionally congruent with the model, minimizing the need for weight adjustment during fine-tuning."},
+{"type": "english_instruction", "instruction": "Write a technical note on the tokenization and deduplication strategies used in Dolma, based on 2025 documentation.", "input": "", "reasoning": "Dolma uses MinHash deduplication with 13-gram shingles and 16 hash functions per band. Tokenization uses GPT-NeoX-20B tokenizer with 50,257 vocabulary size.", "output": "Dolma implements per-document deduplication using MinHash with 13-gram shingles, 16 hash functions per band, and Jaccard similarity threshold of 0.8. This removes near-duplicate documents while preserving legitimate similar content. For tokenization, Dolma uses the GPT-NeoX-20B tokenizer with a vocabulary size of 50,257 tokens, applying byte-level byte-pair encoding (BPE). The corpus also includes URL-based filtering to remove adult content, language identification using fastText with confidence threshold 0.8 (English only), and heuristic filters removing documents with excessive repetition or outlier lengths."},
+{"type": "english_instruction", "instruction": "Write a technical description of DCLM (Data Compaction for Language Models) and its integration with ProX for scalable refinement.", "input": "", "reasoning": "DCLM is a framework for compacting training data by removing redundancy. ProX provides sample-wise cleaning before compaction, ensuring DCLM operates on pristine text.", "output": "DCLM (Data Compaction for Language Models) is a framework that reduces pretraining data size while preserving informational content. It identifies and removes redundant passages within and across documents using semantic similarity hashing and perplexity-based importance scoring. DCLM can reduce dataset size by 50-80% while maintaining or improving downstream task performance. ProX integrates with DCLM as a preprocessing step: ProX first performs sample-wise cleaning (HTML stripping, normalization, anomaly removal) on raw documents. The cleaned output is then fed to DCLM for compaction. This two-stage pipeline ensures that DCLM operates on clean, standardized text, preventing artifacts from being mistakenly identified as informative content and retained."}