Carry Memory Lab

Why can a better bit score still leave the sum wrong?

When I add numbers on paper, I have to remember a carry from one column to the next. This small saved experiment makes that dependency visible. A trained recurrent neural network reads two numbers one bit at a time, starting at the low end. The page runs it twice: once keeping its hidden state, and once resetting that state before every bit. Both modes use the same 106 trained parameters.

The default example is 1 + 15. At width 6, the recurrent mode gives 16 and the reset mode gives 14. Padding the same numbers to width 64 still leaves the reset answer at 14, yet its bit accuracy rises from 3/7 to 61/65 because the extra leading zeros add many easy bits. The whole answer remains wrong. The page keeps the exact integer reference, every output bit and all eight hidden coordinates visible.

The one training run and independent acceptance were completed in the November 4–5 frozen batch. This release publishes that saved experiment; it adds no training run. The ordinary recurrent mode passes all 64,856 fixed operand/width cases, while the reset mode has 58,566 wrong integers. Those are dependent views and finite probes, not a proof that the network learned unrestricted addition. Resetting state also changes the network's input distribution; it is not an optimized memory-free baseline. No hidden coordinate is labelled a carry neuron.

Try the default, change the width while holding the numbers fixed, or enter canonical decimal strings for larger integers. Download the complete two-mode report to inspect logits, probabilities, hidden states, exact answers and errors. Use exact arithmetic for real calculations; this is a teaching experiment. Earlier development/freeze statements below describe the original batch, not a claim that today's public release is still awaiting its first upload.

为什么逐位正确率提高了,加法答案还是错的?

纸上做加法时,我得把上一列的进位记到下一列。这个已保存的小实验把这件事变得可查看:一个循环神经网络从低位开始逐位读取两个数,页面分别保留它的隐藏状态,或在每一位前把状态清零。两种模式共用同一组106个训练参数。

默认例子是1 + 15。位宽6时,保留状态的答案是16,清零模式是14。把同样两个数补零到位宽64,清零答案仍是14,但逐位正确率从3/7提高到61/65,因为多出来的高位零提供了许多容易答对的位置。整体答案仍错。页面同时显示精确整数参照、每一位输出和全部八个隐藏坐标。

这一次训练及独立验收属于11月4日至5日的冻结批次,今天发布保存实验,没有新增训练。保留状态模式通过全部64,856个固定操作数/位宽案例;清零模式保留全部58,566个错误整数。这些视图相互依赖,有限案例通过不证明掌握任意整数加法。清零还改变了网络面对的分布,也不是经过优化的无记忆基线;没有把任何隐藏坐标命名为进位神经元。

可以保持数字不变只改位宽,或用规范十进制字符串输入大整数,下载包含两种模式、完整输入、逐步输出、隐藏状态和错误的报告。真实计算请用精确整数运算。下方原批次开发/冻结说明保留其历史语境,不代表本次公开后仍未上传。

Inspect how a trained RNN adds numbers when its recurrent state is retained or reset. Exact integer addition remains the reference.

I trained one 106-parameter tanh RNN on least-significant-bit-first inputs of widths 1–6. The ordinary network passes the fixed probes below. Resetting its hidden state before every bit produces many wrong integers. These are two inference modes of the same weights, with one actual training run. There is no pretrained model, teacher forcing, or true carry input.

Addition and the carry algorithm are established mathematics. This is an educational study of a learned approximation in a designed representation. Finite passing probes do not prove unrestricted addition. Interpretation limits.

Results and denominators

Inventory Cases per mode Recurrent: correct integers Reset: correct integers Recurrent bit accuracy Reset bit accuracy
Complete unique operand/width cases 64,856 64,856 6,290 (9.6984%) 100% 59.5060%
Held-out, new-range and padding union 64,499 64,499 6,114 (9.4792%) 100% 59.6801%

The complete inventory includes 17 exact training representations in its stress suites, explicitly tagged. The primary union contains the declared held-out, new-range and padding suites. Suite memberships overlap: 64,882 memberships refer to 64,856 unique ordered operand/width cases. Reverse pairs, padding views and repeated modes are dependent; they are not extra training runs or independent arithmetic problems.

All 58,566 integer failures remain. Neural answers are never replaced by the reference. The default 1 + 15, width 6, was chosen by its exact carry pattern in a held-out group before model testing. The recurrent model predicts 16; reset predicts 14. At width 64, reset still predicts 14, while its bit score rises from 3/7 to 61/65. Padding can improve bit accuracy without fixing the integer.

Training, suite results, padding control and carry errors

Open the full-size state figure. The figures include all 80 epochs, nine suites, 500 padding views, 127 carry probes, and both modes' default hidden states. No hidden coordinate is identified as a carry neuron.

Complete artifacts:

Run the browser demo

Serve this folder through static HTTPS or loopback. Opening file:// directly is unsupported.

python3 -m http.server 8000 --bind 127.0.0.1

The page uses native browser modules, Web Crypto, BigInt and structuredClone. There is no build step, server inference, external model API, font download, analytics or browser storage. Model and fixed-result bytes are checked before the controls activate.

Enter canonical decimal strings from 0 through 18446744073709551615, with a width from 1 through 64 that fits both operands. The sum may need 65 bits. Operands and decoded integers never pass through JavaScript Number. JSON requires exactly a, b and width:

{"a":"9007199254740993","b":"2","width":54}

Signs, decimal points, exponents and leading zeros are rejected for operands. Blank inputs do not become zero. JSON uses JSON.parse last-key semantics and preserves the raw text. Failed inputs clear the current report. Explicit field editing restores field mode; changing the hidden-state view cannot revive an old request.

The page shows both modes, the exact reference, every bit and all eight hidden coordinates. Tables have keyboard focus and horizontal scrolling on small screens. Downloads preserve both modes, full inputs, all steps, logits, log probabilities, hidden activations, wrong positions, grouping/padding flags, hashes and attribution. NLL comes from stable log-softmax, without probability clipping or calibration.

Inputs remain in page memory. Clear or refresh removes current edits; deliberately downloaded files remain where saved. Use exact arithmetic for real calculations.

Data and fixed training plan

Protocol, configuration, research basis, and data audit. Examples are independently generated from the standard integer rule. Original code, example packaging, weights, results and figures are Apache-2.0, Renchao Wu. No external dataset or pretrained weights are used.

The method reference is the D2L author textbook's BPTT chapter, read for principles. No textbook code, prose or figures are copied. This implementation uses tanh and biases; it is not an exact reproduction of the chapter's simplified identity/no-bias example.

The small inventory exhausts all 5,460 ordered pairs represented at widths 1–6. Its 2,080 unordered numeric-pair groups exclude width and direction from their keys. A fixed SHA split assigns whole groups to:

Role Representations Unordered groups
Fit 3,847 1,460
Validation 554 210
Test 1,059 410

Reverse and leading-zero representations cannot cross roles. Validation has no width-1 examples; support is not invented. Small inventory, complete evaluation cases.

The evaluation includes exhaustive new operand ranges at widths 7/8 (12,288/49,152 cases), fixed random new-range cases at widths 16/32/64 (500 each), 127 carry-chain probes, 256 zero/maximum edges, and 500 padding views. Wide inputs change magnitude and length together. Padding holds values fixed. Stress overlaps and known pairs remain tagged.

One RNN has two inputs, eight tanh hidden coordinates, an 8×8 recurrent matrix and two softmax outputs: 106 parameters with biases. Each sequence starts from zero, consumes only current operand bits and previous hidden state, then receives a terminal (0, 0) input. Seed 329, 80 full-BPTT epochs, batch 128, 2,720 updates, 307,760 representation visits and 2,052,000 bit visits. Adam uses learning rate .01, betas .9/.999 and epsilon 1e-8. Weight L2 is .0001; global gradient clipping at 1.0 was configured but triggered zero times.

Checkpoint 80 is the earliest minimum validation bit NLL. There was no architecture/seed search or post-test tuning. Trace, training metadata, model. Protocol/training freezes precede fitting; PRETEST binds the selected artifacts before the separately frozen evaluator runs.

Reproduce on a copy

Checked with Python 3.14.7, NumPy 2.5.4 and Node 24.19.0. Optional plots use Matplotlib 3.11.2. Copy the candidate first: verifiers write artifacts in their working directory.

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test_core.py
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python verify_results.py
node test_portable.mjs
node test_ui_races.mjs

The Python oracle checks every source group, target, suite, metric, failure and mode-bit against an independent scalar recurrence. It caches only identical complete input histories. It generates portable-prefix-reference.json.gz (about 21 MB), a reproducible cache excluded from the release. Run it before the Node test. Node then checks all history states and all 64,856 complete BigInt requests.

Thirteen final native reports cover ten distinct input/width cases; the final form-isolation check deliberately repeats a field-recovery input. Two full examples and compact evidence are included; the full parent evidence is retained for review. verify_native.py requires the thirteen native files and actual UI records from the parent directory. It does not replace a reviewer's browser check. Validation details.

Fresh fitting and evaluation use new output directories:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 PYTHONDONTWRITEBYTECODE=1 .venv/bin/python train.py --out fresh-run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python evaluate.py --run fresh-run --out fresh-evaluation

The synthetic gradient test writes gradient-results.json beside its script. On a different numerical platform, run that test from a separate core.py/test_core.py copy to preserve the author's frozen gradient record before training. Record fresh hashes and compare values; do not edit accepted author gates. Cross-platform byte identity is not promised.

Identity: work:carry-memory-lab:v1, version 0.1.0. Local checks are separate from independent acceptance, public deployment and adoption. This batch remains development/review/freeze only; later releases use the original daily HF workflow.