Writing AI Papers and Submitting to arXiv: A Practical Guide to Structure, Experiments, and Conference Strategy

🌏 中文版

TL;DR

Writing an AI paper and submitting it to arXiv involves four layers of work, all covered here:

1. Paper structure — what each section from Abstract to Limitations should and should not contain
2. Experimental design — dual-check with the NeurIPS Reproducibility Checklist (16 items) + REFORMS (8 modules, 32 items)
3. arXiv submission — the 2026 endorsement update, category selection, licensing, version management
4. Conference vs. direct arXiv — the latest preprint policies of five top conferences and a recommended strategy

Every claim is traceable to an official source. Each section ends with a ready-to-use checklist.

The Context

You likely have hands-on experience with RAG systems or agents. You've read hundreds of papers. You've started thinking, "This improvement is worth writing up." But between implementation and a submitted paper there's a fuzzy gap — how complete must the experiments be? How many baselines? Should you release code? Which arXiv category?

This is not an academic writing textbook (there are plenty of those). It's a practical checklist from an AI/ML engineer's perspective, laying out everything you need to check before you submit.

Paper Structure: What Each Section Needs (and What to Avoid)

The standard AI/ML paper structure is well-established (all top conference author kits specify it). The common problem isn't a missing section — it's the wrong content inside it.

Abstract

Goal: Let readers decide in 30 seconds whether to read the full paper.
Content: Problem statement → core idea → main results (with numbers) → 1-2 contribution sentences.
Pitfalls:

• Describes the method without reporting results — if you don't tell me how well it works, why should I read on?
• States motivation without the solution — what did you actually do?
• Hyperbolic language ("first-ever", "significant breakthrough") — reviewers will check this against your actual results

Introduction

Goal: Establish that this is a real problem, existing methods fall short, and your approach is both reasonable and effective.
Content: Problem background → limitations of existing work (cited) → your approach in 2-3 sentences → contribution list → paper organization.
Pitfalls:

• Related work content leaking into the Introduction — makes it bloated and hard to read
• Contributions framed as a feature list — "We propose framework X" isn't a contribution; "Framework X beats Z by 5.2% on benchmark Y" is

Related Work

Goal: Position your work in the literature and demonstrate your awareness of the field.
Content: Grouped by topic (not chronological enumeration). Discuss representative methods and their limitations for each group.
Pitfalls:

• Listing without commentary — a reference list disguised as a section, which signals a superficial understanding
• Claiming "nobody has done this" for something that has been published — reviewers in that subfield will catch this immediately

Method

Goal: Enable a graduate student in the field to fully reproduce your approach.
Content: Formal problem definition → model architecture (with diagram) → training objective → inference procedure → key implementation details.
Pitfalls:

• Missing implementation details — learning rate, batch size, optimizer, warmup, dropout are considered "common knowledge" but missing one can cost 3-5 percentage points in reproduction
• Diagrams that are too simple or too complex — a good diagram should convey the architecture even without reading the text

Experiments

This is the most critical section and the one with the most problems.

Goal: Prove that your method beats the baselines, and that you understand why.
Content: Setup (datasets, metrics, implementation details) → main results (tables/figures) → analysis (ablation, visualizations, case studies).

Key results table placement: The NeurIPS 2026 author kit requires the key results table to appear in the main paper, not the appendix. This is common across top conferences — reviewers rarely consult appendices during the rebuttal phase, so a key table hidden in the appendix is effectively absent.

Pitfalls (each one is a red flag):

• Baselines more than 2 years old — the field moves fast; stale baselines signal cherry-picking
• Testing only on self-curated datasets — makes fair comparison impossible
• Reporting only the most favorable metric — selective reporting
• No error bars — results could be random noise
• Baselines run with default parameters while your method is carefully tuned — the most common unfair comparison
• Key results table in the appendix — effectively invisible to reviewers

Limitations

Goal: Honestly discuss the boundaries of your method.
Content: Scenarios where performance degrades, theoretical limitations, directions for improvement.
Pitfalls:

• Framing limitations as future work — "Can be applied to domain X" is not a limitation
• Omitting the section entirely — this has become a red flag; reviewers will question your self-awareness

Broader Impact

Required by NeurIPS since 2020. Goal: Discuss potential positive and negative societal impacts.
Content: Potential for misuse, bias issues, environmental costs.
Pitfalls:

• Only listing positives — signals superficial consideration
• Vague generic statements — "May impact society" is as good as saying nothing

Experimental Design Standards: Two Checklists You Must Run

NeurIPS ML Reproducibility Checklist

Required since 2022. The 2026 version has evolved into 16 conditional questions (each answer may reveal follow-up items):

The full checklist is at the NeurIPS Paper Checklist Guidelines. Answer every question before submission; don't just tick the box.

REFORMS Framework (arXiv:2308.07832)

REFORMS (Reporting Standards for ML Based Science) was developed by 19 researchers at Princeton and published in Science Advances in 2024 (DOI: 10.1126/sciadv.adk3452). It targets ML applied to scientific discovery (not pure methodology papers). It has 8 modules and 32 items:

Module 1: Study Goals (3 items)

• 1a: Population or distribution about which the scientific claim is made
• 1b: Motivation for choosing this population
• 1c: Motivation for using ML methods in the study

Module 2: Computational Reproducibility (5 items)

• 2a: Permanent link or DOI to the exact dataset version
• 2b: Commit tag or DOI for the code
• 2c: Computing infrastructure (hardware, OS, package versions)
• 2d: README with complete reproduction steps
• 2e: Reproduction script that produces all results

Module 3: Data Quality (7 items)

• 3a: Data source details (time period, location, collection/annotation process)
• 3b: Sampling frame and method
• 3c: Justification for why this dataset suits the modeling task
• 3d: Outcome variable definition and descriptive statistics
• 3e: Sample counts (total + per class)
• 3f: Missing data percentage (stratified by class)
• 3g: Whether the evaluation dataset is representative of the population

Module 4: Data Preprocessing (3 items)

• 4a: Excluded data and rationale
• 4b: How impossible or corrupted samples are handled
• 4c: Complete sequence of all data transformations (imputation, normalization, augmentation) — data-dependent transformations must be done after splitting

Module 5: Modeling (6 items)

• 5a: Model description (inputs, outputs, type, loss function)
• 5b: Justification for the chosen model type
• 5c: Evaluation method (cross-validation, held-out, external)
• 5d: Model selection method
• 5e: Hyperparameter selection (search range + final values)
• 5f: Whether baselines are appropriate and fairly tuned

Module 6: Data Leakage (3 items)

• 6a: Whether train-test separation is maintained (preprocessing and modeling use only training data)
• 6b: Whether there are dependencies or duplicates between train and test sets
• 6c: Whether each feature is legitimate for the task, not a proxy for the outcome

Module 7: Metrics and Uncertainty (3 items)

• 7a: All performance metrics used, including intermediate decision metrics
• 7b: Uncertainty estimates for each metric (standard deviation, CI, bootstrap)
• 7c: Justification for statistical tests (if used)

Module 8: Generalizability and Limitations (2 items)

• 8a: External validation results (or acknowledgment of absence)
• 8b: Contexts in which findings are not expected to hold

The full REFORMS checklist is at reforms.cs.princeton.edu.

Using Both Checklists Together

The NeurIPS Checklist is your minimum bar — answer every question before submitting. REFORMS is a more detailed scientific reporting standard. If your paper is about ML applied to scientific discovery (rather than pure methodology), run both.

Practical advice: Start with the NeurIPS 16 questions to catch basic gaps, then use the REFORMS 8 modules for depth. If more than 3 items remain unchecked (per CodeSOTA's recommendation), treat the results as "preliminary and unverified."

arXiv Submission: From Endorsement to Versioning

Endorsement: The 2026 Update

arXiv has required endorsement for first-time submitters since 2004. On January 21, 2026, arXiv updated the policy: an institutional email alone is no longer sufficient.

New submitters now need one of the following:

1. Institutional email + prior authorship in the same domain: Both an academic/research email address AND a record of co-authorship on an existing arXiv paper in the same domain (claimed through the arXiv authority system)
2. Personal endorsement: An endorsement from an active arXiv author in the same domain, who must have a paper in that domain from 3 months to 5 years ago

For graduate students, the easiest path is to ask your advisor — as long as they have a paper in the relevant domain on arXiv.

Endorsement is domain-specific — an endorsement in cs does not apply to math.

Category Selection

arXiv categories are hierarchical: major (cs, math, stat) → subcategories. AI/ML-related subcategories:

Recommendation: Choose one primary category and cross-list 2-3 related ones. Moderators may reclassify if they deem the category inappropriate (being moved to cs.general is considered a downgrade in the community).

License Selection

arXiv offers several license options. The choice is irrevocable:

Practical advice: Most people use the arXiv non-exclusive license. If you're also submitting to a conference, check their preprint license requirements first.

Version Management

Common Moderation Rejection Reasons

arXiv moderators are volunteer domain experts with terminal degrees. About 6% of submissions are held and about 2% are rejected (Scientific American). Common reasons:

• Format violations (line numbers, watermarks, ads, margin notes, referee remarks)
• Non-scholarly content (coursework, project proposals, news commentary)
• Plagiarism or falsified data
• Offensive images (violence, pornography, the controversial "Lena" image)
• Undisclosed significant generative AI use
• AI listed as an author
• Excessive submission rate (max 3 per day)
• Copyright conflicts (copyright statements prohibiting arXiv distribution)

HTML Generation

Since December 2023, arXiv auto-generates HTML versions for all TeX/LaTeX submissions (using NIST's LaTeXML). HTML offers far better readability and accessibility than PDF.

Author recommendations: Use LaTeXML-supported packages (GitHub list), avoid packages with limited support like tikz, add alt text to images, and set Overleaf's compiler to "stop on errors."

Conference Submission vs. Direct arXiv: Policy Differences Across Five Top Venues

This is one of the most confusing questions for first-time submitters. arXiv cannot replace conferences, and conferences should not replace arXiv. All top conferences now allow arXiv preprints alongside submissions, but with varying restrictions.

2024-2026 Preprint Policy Comparison

Key update: ACL made a major change in January 2024 — completely eliminated the anonymity period. Authors can post non-anonymous preprints at any time. Reviewers must disclose outside knowledge, and tie-breaking favors anonymous submissions. This is currently the most open policy.

arXiv vs. Conference Timeline

Recommended Strategy: Simultaneous Submission

For most papers, the optimal strategy is simultaneous conference + arXiv submission:

Day 0:       Submit to conference + post arXiv v1 (same version, anonymized for double-blind)
Review:      Incorporate community feedback → arXiv v2
Rebuttal:    Address reviewer concerns → arXiv v3 (if timing allows)
Acceptance:  Post camera-ready as final arXiv version (respect embargo)
Post:        Add conference DOI to arXiv Comments field

arXiv's timestamp provides effective scooping protection. The SiLU function (2016 arXiv) vs. Swish (2017 Google) is a classic case — the arXiv timestamp established priority (Paul Ginsparg).

When to Only Post to arXiv

• Research that needs rapid dissemination (security flaws, new benchmarks, negative results)
• Missed conference deadlines
• Work that is a survey, replication, or preliminary study
• No career dependency on formal publication

Rebuttal: Practical Principles for Responding to Reviewers

Conference rebuttals typically happen within a week of receiving reviews, with character limits (NeurIPS: per-reviewer limits; ICML: 5000 total). The principles below are drawn from official conference guidelines and experienced researchers' practical advice.

Structure

All comments from Reviewer X
└─ Q1: "Unclear design choice for Y"
   └─ Thank the reviewer → clarify the original design, reference paper sections
   └─ (If necessary) add supplementary experimental results
└─ Q2: "Why not compare with Z"
   └─ Acknowledge this as a good suggestion → explain Z's applicability differences
   └─ (If time permits) add the experiment

Core principles (summarized from Niklas Elmqvist, Matt Might, and others):

• Respond to every comment — skipping one guarantees the reviewer will flag it
• Don't fight the reviewer — "You're wrong, our paper explains this clearly" is a disaster in a rebuttal
• Distinguish clarify from experiments — when misunderstood, prioritize clarification; when experiments are needed, be honest about whether you can complete them within the rebuttal window and commit to adding them in camera-ready
• Don't overstate — "We've completely solved this problem" will make the reviewer feel you didn't understand their criticism

Camera-Ready Preparation

You typically have 3-5 weeks between acceptance and the camera-ready deadline. Main tasks:

• Page limit: NeurIPS/ICML allow +1 page; ICLR same as submission
• Author list: Restore from the anonymous version (some conferences restrict additions, e.g., CVPR 2026 allows at most 1 new author)
• NeurIPS Checklist: Upload alongside the main paper
• Video: ICLR and some others require a 5-minute presentation video
• Code submission: Some conferences encourage or require code
• Syncing with arXiv: Push the camera-ready version to arXiv — watch for any embargo period

Putting It All Together

Writing an AI paper and successfully submitting it is a multi-stage engineering process. Each step — structure, experiments, review, versioning — has its own specification. The most efficient strategy is to align all checklists before you start writing, rather than going back to fill gaps after the paper is drafted.

The core proposition: The NeurIPS Reproducibility Checklist is your minimum bar, REFORMS is your deep check, the arXiv official docs are your submission manual, and the conference policies are your strategic framework — all four are indispensable.

References

arXiv Official Documents

• The arXiv endorsement system
• Attention Authors: updated endorsement policy (Jan 2026)
• arXiv Category Taxonomy
• arXiv License Information
• Submission Version Availability
• To replace an article
• Withdrawing an article
• arXiv moderation
• LaTeX Markup Best Practices for Successful HTML Papers
• HTML papers on arXiv: why it's important, and how we made it happen (Feb 2024)

Reproducibility and Reporting Standards

• NeurIPS Paper Checklist Guidelines
• REFORMS: Reporting Standards for ML Based Science (arXiv:2308.07832, Science Advances 2024)
• REFORMS Official Website
• ML Reproducibility Checklist 2026 (NeurIPS)
• Questionable Practices in Machine Learning (arXiv 2407.12220)
• Princeton Reproducibility Crisis in ML-based Science
• CodeSOTA — How to Read an ML Paper

Conference Submission Policies

• NeurIPS Call for Papers 2026
• ICML 2026 Author Instructions
• ICLR 2026 Author Guide
• CVPR 2026 Author Guidelines
• ACL Anonymity Policy (2024 Revision)

Citation Impact and Conference vs. arXiv Research

• Feldman, Lo, Ammar (2018) — 65% citation advantage for arXiv-first (arXiv:1805.05238)
• Elazar et al. (2024) — Causal effect of arXiving on acceptance <4% (arXiv:2306.13891)
• Mishkin, Tabb, Matas (2020) — arXiving helps everyone (arXiv:2010.05365)
• Gloire & Mitra (2026) — LLM de-anonymization 42% accuracy (arXiv:2608.05157)

Rebuttal and Writing Guides

• How to Read a Paper — S. Keshav (ACM SIGCOMM CCR, 2007)
• Ten Simple Rules for Writing a Response to Reviewers — William Stafford Noble (PLOS CB, 2017)
• Writing Rebuttals — Niklas Elmqvist (Medium, 2016)
• How to Write a Rebuttal Letter — Matt Might
• Site: How to Read arXiv Papers: Methodology and Tool Landscape
• Site: arXiv Paper Quality Assessment Guide
• Site: How Others Use LLMs for Writing (zh-TW only)