XGBoost subsample & sampling_method: The Bagging Lever That Quietly Saves Your Regression

By Shakti Tiwari — Nifty Option Trader, XGBoost Expert (optiontradingwithai.in)

Disclaimer: Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Quick Answer

XGBoost's subsample controls the fraction of training rows fed to each boosting round — it is row-level bagging applied per tree. In our observed 5-fold CV, leaving it at the default 1.0 was the worst setting for regression (RMSE 0.4611), while 0.6 won with RMSE 0.4057 — a ~13.7% error reduction. Classification accuracy barely moved (0.9063→0.9072 across all values). The companion sampling_method (uniform vs gradient_based) decides how rows are picked: uniform draws equally, gradient_based oversamples hard-to-fit rows. Tune subsample in the 0.6–0.9 band, never blindly keep 1.0.

Why This Matters

Most people tune max_depth, eta, and lambda and then walk away. They never touch subsample. Big mistake — especially if you trade or forecast.

Here is the deal. A gradient-boosted tree model adds trees one at a time, each one correcting the residual of the last. If every tree sees all the rows (subsample = 1.0), the trees become highly correlated — they all learn the same quirks of the same data. That is variance you are carrying into production, and variance is exactly what blows up your out-of-sample numbers on expiry day.

subsample breaks that correlation. Each tree trains on a random row-draw, so the ensemble behaves a bit like a bagged model: individual trees differ, their errors partially cancel, and generalization improves. In our data, that single change from 1.0 → 0.6 chopped regression error by double digits. That is not a rounding artifact; it is the difference between a model you trust and a model you paper-trade-and-pray.

And yes, this matters for the two-layer engine we run. Our live system is TWO-LAYER: Layer 1 = Dhan WebSocket live capture (shadow/predict-only, never trades without broker auth); Layer 2 = EOD-audited XGBoost/LightGBM training core (walk-forward, cost-and-slippage model). The parameter discussed is tuned in Layer 2. So subsample is a Layer-2 knob, tuned on end-of-day audited folds — not something flipping around on the live tick stream.

Samajh lo: subsample is your cheapest regularization you are probably not using.

Research Question & Hypothesis

Question: Across a fixed 5-fold CV harness, how does subsample ∈ {0.4, 0.6, 0.8, 0.9, 1.0} move (a) regression RMSE and (b) classification accuracy, holding all other params constant?

Hypothesis: Smaller subsample → more row-level stochasticity → lower variance → better OOS, up to a point. Below some threshold the trees starve (too little data per round) and underfit, so we expect a U-shaped (or at least non-monotonic) curve with a sweet spot, not a straight line.

We also wanted to check a second, subtler claim: that row subsampling behaves differently for regression vs classification. The data answered that loudly — see Results.

Data & Methodology Box

The harness is deterministic (global RNG seed 42, fold seed 0). The numbers below are observed, not theoretical, not interpolated.

Results

Observed 5-fold CV table

Findings (3–7)

1. The default subsample=1.0 is the worst regression setting here. RMSE 0.4611 is ~13.7% higher than the 0.6 optimum (0.4057). "Default = safe" is false for this lever.

2. Sweet spot is ~0.6. Reg RMSE bottoms at 0.4057 for subsample 0.6, with 0.8 close behind (0.4194, +3.4%). The good band is roughly 0.6–0.8.

3. Too small hurts — underfitting is real. Dropping to 0.4 raises RMSE back to 0.4140 (+2.0% vs 0.6). Each tree sees only 40% of rows, so it can't capture the X0²/sin(X1) structure cleanly. Confirmed non-monotonic, as hypothesized.

4. Classification accuracy is essentially flat. Spread is only 0.9063 → 0.9072 (a 0.0009 window). Row subsampling is not a classification-accuracy knob on this dataset — it is a regression-generalization lever.

5. Metrics disagree at the margin. subsample 0.9 gives the best clf ACC (0.9072) but the third-worst reg RMSE (0.4233). Tune to the metric you actually deploy on — don't average-rank across objectives blindly.

6. The curve is U-shaped, not linear. 1.0 (bad) → 0.6 (best) → 0.4 (worse again). Smaller-is-better intuition fails below the knee.

Reproducibility (Code)

The exact sweep, runnable as-is (takes ~minutes for one param; the full A2–A7 run is ~50 min):

import numpy as np, xgboost as xgb, json

def make_reg(n=20000, d=24, seed=1):
    r = np.random.default_rng(seed)
    X = r.normal(0, 1, size=(n, d))
    y = (X[:,0]**2 + np.sin(X[:,1]) + X[:,2]*X[:,3]
         + 0.5*X[:,4] - 0.3*X[:,5] + r.normal(0,0.3,n))
    return X, y

def make_clf(n=20000, d=24, seed=2):
    r = np.random.default_rng(seed)
    X = r.normal(0, 1, size=(n, d))
    score = (X[:,0]*0.8 + X[:,1]**2*0.5 - X[:,2]*0.6
             + np.tanh(X[:,3]) + r.normal(0,0.4,n))
    y = (score > 0).astype(int)
    return X, y

def kfold(X, y, n_split=5, seed=0):
    idx = np.arange(len(y)); r2 = np.random.default_rng(seed); r2.shuffle(idx)
    parts = np.array_split(idx, n_split)
    for i in range(n_split):
        te = parts[i]; tr = np.concatenate([parts[j] for j in range(n_split) if j!=i])
        yield tr, te

Xr, yr = make_reg(); Xc, yc = make_clf()
results = {}

def oos_reg(param, values, fixed):
    out={}
    for v in values:
        p = dict(max_depth=4, eta=0.1, subsample=0.8, colsample_bytree=0.8,
                 objective='reg:squarederror', verbosity=0)
        p.update(fixed); p[param]=v
        oos=[]
        for tr,te in kfold(Xr, yr):
            dtr=xgb.DMatrix(Xr[tr], label=yr[tr]); dte=xgb.DMatrix(Xr[te], label=yr[te])
            bst=xgb.train(p, dtr, num_boost_round=500, evals=[(dte,'te')],
                          early_stopping_rounds=30, verbose_eval=False)
            pred=bst.predict(dte, iteration_range=(0, bst.best_iteration+1))
            oos.append(np.sqrt(np.mean((yr[te]-pred)**2)))
        out[str(v)]=round(float(np.mean(oos)),4)
    return out

## Same shape for oos_clf with objective='binary:logistic', metric = accuracy
results['A5_sub_reg'] = oos_reg('subsample', [0.4,0.6,0.8,0.9,1.0], {})
## results['A5_sub_clf'] = oos_clf('subsample', [0.4,0.6,0.8,0.9,1.0], {})
json.dump(results, open('a5_sub.json','w'), indent=2)

To switch on sampling_method (the companion knob), the scikit-learn API is cleaner:

import xgboost as xgb

## Default uniform row-draw, the setting used in the sweep above
reg_uniform = xgb.XGBRegressor(
    n_estimators=500, max_depth=4, eta=0.1,
    subsample=0.6, colsample_bytree=0.8,
    sampling_method='uniform', early_stopping_rounds=30)

## Gradient-based: oversample hard rows (needs hist/gpu_hist tree method)
clf_grad = xgb.XGBClassifier(
    n_estimators=500, max_depth=4, eta=0.1,
    subsample=0.8, colsample_bytree=0.8,
    sampling_method='gradient_based', tree_method='hist',
    early_stopping_rounds=30)

Run it, you will reproduce the table. Garbage in, garbage out — but here the numbers are pinned.

Deep Dive: sampling_method — Uniform vs Gradient-Based

subsample answers how many rows; sampling_method answers which rows. Per doc/parameter.rst there are two modes, and they change the character of the stochasticity completely.

uniform (default). Every row gets the same probability subsample of being drawn into a given boosting round. It is classic bootstrap-style bagging: the draw is blind to how wrong the model currently is. This is what the sweep above used, and it is the right default when you just want variance reduction without assumptions about your data.

gradient_based. Here the inclusion probability of row i is proportional to the magnitude of its gradient |g_i| — i.e., how large the current residual is. For reg:squarederror the gradient is literally the negative residual (y_i − ŷ_i), so a row the model is currently missing by a mile is far more likely to be sampled than a row it already fits tightly. The effect: each new tree is steered toward the hard-to-fit rows. It is closer in spirit to boosting's own logic (focus on residuals) but applied at the row-draw level rather than the tree-weight level.

Two practical constraints straight from the docs and source:

• gradient_based only takes effect when subsample < 1. At subsample=1.0 there is no draw to reweight, so the flag is inert.
• It requires tree_method='hist' or 'gpu_hist'. The gradient-weighted sketching is built on the histogram tree builder; the exact ('exact') and approximate ('approx') methods don't support it. In practice that just means: if you want gradient_based, set tree_method='hist' (or let XGBoost default to it on modern versions).

When should you reach for gradient_based over uniform? Think imbalanced difficulty. If 5% of your rows carry 50% of the residual (fat tails in option IV surfaces, rare volatile sessions, a minority class that's genuinely harder), uniform keeps under-sampling those stubborn rows by chance, while gradient_based deliberately over-represents them. The trade-off: it can over-focus on noise if those large gradients are just stochastic outliers, so validate it the same way we validated subsample — out-of-fold, on the metric you deploy.

How to Read the Subsampling Curve (A Walk-Through)

Numbers on a page are cold. Here is how I actually read the table above, step by step, so you can do it on your own data.

1. Anchor on the default. Start at subsample=1.0, RMSE 0.4611. That is your "no bagging" baseline. Everything is measured relative to it.
2. Walk left. 0.9 → 0.4233 (better). 0.8 → 0.4194 (better still). 0.6 → 0.4057 (the floor). The error is falling as we drop rows — variance is being taxed away.
3. Past the knee. 0.4 → 0.4140. Error rises again. We crossed from "just enough stochasticity" into "not enough data per tree." That uptick is your underfit warning light.
4. Find the valley, not the edge. The minimum is at 0.6, not at the smallest tested value (0.4). A common amateur mistake is to see "smaller = better so far" and pick 0.4 — but the curve has already turned. Always plot the full sweep; never tune by hill-climbing one step.
5. Check the second metric. Classification ACC is flat the whole way (0.9063–0.9072). So if your deploy metric were accuracy, subsample would be a free parameter you could leave near 1.0 to save compute. Tuning budget is finite — spend it where the metric moves.
6. Note disagreements. 0.9 is best for clf but mediocre for reg. If you ever multi-objective rank, weight by deploy priority, don't average naively.

That six-step read generalizes to every continuous XGBoost knob: anchor, sweep, find the valley, respect the knee, cross-check the second metric, and rank by deploy weight. The shape here — a clean U — is the textbook signature of a well-behaved regularization parameter.

What Failed / Counter-Evidence

We went in expecting subsample to lift both metrics. It didn't. The classification column stayed boring-flat the entire sweep. That is counter-evidence to the "bagging helps everything" mental model.

Why might that be? In this synthetic clf task the decision boundary is fairly smooth and the label noise (N(0,0.4) on the score) is symmetric. Row-dropping barely changes which side of the threshold a point lands on, so accuracy is stable. Regression, by contrast, is sensitive to how well the continuous surface is fit — and correlated trees (subsample 1.0) overfit that surface's wiggles, inflating RMSE. So the "failure" is actually the most instructive result: subsample is a variance-tax you pay where the metric punishes variance — regression more than balanced classification.

We also half-expected 0.4 to beat 0.6. It didn't — 0.4 underfits. Good to know; it kills the "more stochasticity = always better" story.

Limitations

• Synthetic data. The targets are built from known nonlinearities + Gaussian noise. Real Nifty option-chain features are fatter-tailed, gappy, and regime-dependent. The shape of the curve (U-shaped, default-worst) is the transferable lesson; the exact 0.6 optimum is not a promise on your data.
• Single fixed config. max_depth=4, eta=0.1, colsample_bytree=0.8 are held constant. subsample interacts with colsample_* (next article, A6) and eta — a deeper tree or different learning rate would shift the optimum.
• sampling_method not swept here. This run used uniform. gradient_based is a separate axis we describe but did not CV in the table.
• One seed family. Fold seed 0, global 42. With 20k rows and 5 folds the estimate is stable, but we did not bootstrap confidence intervals.

Practical Takeaways (Production Checklist)

1. Never ship subsample=1.0 untested. Make 0.6–0.9 a mandatory grid point. Our data shows the default can cost you ~14% regression error.
2. Tune to your deploy metric. If you deploy on RMSE/MAE, center the search on 0.6–0.8. If on balanced accuracy and your task is smooth like ours, subsample is low-priority — spend tuning budget elsewhere.
3. Watch for underfit. If dropping subsample raises error past a knee, stop — you have starved the trees. 0.4 was already too thin here.
4. Pair with colsample_bytree (A6). Row + column subsampling are complementary stochasticity. Together they decouple trees more than either alone. Don't tune one and freeze the other at 0.8 blindly.
5. Use gradient_based for hard, imbalanced rows. If your loss is dominated by a few stubborn mispredicted rows (class imbalance, fat tails), sampling_method='gradient_based' with tree_method='hist' re-weights the draw toward large-gradient rows — effectively telling XGBoost "spend more trees on where you're currently wrong."
6. Keep subsample in Layer 2. Per the two-layer note, this is an EOD-audited training-core knob, walk-forward validated with the cost-and-slippage model — not a live-tick parameter.

FAQ

Q: Is subsample the same as colsample_bytree?
No. subsample samples rows (training instances) per boosting round; colsample_bytree samples features/columns per tree. Different axes of stochasticity. A6 covers columns.

Q: Can I set subsample > 1.0?
No. Valid range is (0, 1]. 1.0 means "use all rows." There is no over-sampling of rows in subsample; for that you'd look at gradient_based reweighting, not a value > 1.

Q: What does sampling_method='gradient_based' actually do?
Per doc/parameter.rst, instead of drawing rows uniformly, XGBoost weights each row by the magnitude of its gradient (how wrong the current model is on it) and samples proportionally. Hard-to-fit rows get oversampled. It requires tree_method='hist' or 'gpu_hist'.

Q: Why did classification accuracy not move?
Because in this task accuracy is a coarse, threshold-based metric on a smooth boundary; dropping rows doesn't flip many predictions. Regression RMSE, being continuous and variance-sensitive, responds strongly.

Q: Should I use subsample with a small dataset (n=500)?
Carefully. At 0.6 you'd give each tree only 300 rows — risk of severe underfit. The thinner your data, the higher you should keep subsample (0.8–1.0). Our 20k-row result does not transfer linearly to tiny sets.

Q: Does subsample slow training?
Marginally. Sampling rows is cheap; the bigger cost is that lower subsample may need more rounds to converge. With early-stopping it self-limits.

TL;DR

• subsample = row-level bagging per boosting round; default 1.0 is often the worst choice.
• Observed optimum here: 0.6 (reg RMSE 0.4057), ~13.7% better than default 1.0 (0.4611).
• Curve is U-shaped: 0.4 underfits (0.4140), 0.6–0.8 sweet, 1.0 overfits.
• Classification accuracy stayed flat (0.9063–0.9072) — subsample is a regression-generalization lever, not a clf-accuracy lever here.
• sampling_method: uniform (default, equal draw) vs gradient_based (oversamples hard rows; needs hist tree method).
• Tune in Layer 2 (EOD-audited core), in the 0.6–0.9 band, paired with colsample_* (A6).

📚 Related Articles

• A6 Colsample
• A4 Gamma
• A2 Max Depth

👤 About the Author

Name:

Shakti Tiwari

Role:

Nifty Option Trader, XGBoost Expert

Cert:

NISM-Series-XII (Securities Markets Foundation)

Knows:

XGBoost · LightGBM · Options Trading · Machine Learning · Walk-forward Validation

Profiles:
about.me ·
optiontradingwithai.in ·
github ·
whatsapp ·
x/twitter

Sources

• dmlc/xgboost GitHub repository — github.com/dmlc/xgboost (~28.7k★). Primary source for parameter semantics.
• doc/parameter.rst — defines subsample (row subsample ratio, range (0,1], default 1) and sampling_method (uniform / gradient_based).
• doc/tutorials/param_tuning.rst — tuning guidance; recommends subsample + colsample_bytree as stochastic regularization.
• Observed 5-fold CV results: a2_a7_results.json, keys A5_sub_reg / A5_sub_clf (this article's numbers).

Author / Canonical

Shakti Tiwari — Nifty Option Trader, XGBoost Expert
Educational content on optiontradingwithai.in. Not SEBI-registered RA; NISM-Series-XII certified. This article is part of the XGBoost internals hub-and-spoke series (A1…A7). Canonical URL: optiontradingwithai.in/xgboost-subsample.

Resources & Links

• About Shakti: https://about.me/shaktitiwari
• Main site: https://optiontradingwithai.in
• WhatsApp: https://wa.me/919169650895
• Previous in series — XGBoost gamma (A4): https://optiontradingwithai.in/xgboost-gamma
• Next in series — XGBoost colsample_* (A6): https://optiontradingwithai.in/xgboost-colsample
• Book on Amazon (ASIN B0H9ZNTBPK): https://www.amazon.in/dp/B0H9ZNTBPK
• Book on Amazon (ASIN B0HBBFKDQF): https://www.amazon.in/dp/B0HBBFKDQF