AUC
ATIF Holdings Limited
stock NASDAQ

319 changes the whole picture, and it is better news than "I learned how not to use xgboost".
At an out-of-sample n of 319, split roughly evenly between classes, the standard error on an AUC near 0.5 is about **0.032**. That is 3.2 percentage points. The gap you were measuring is 1.6 points. So the thing you are trying to detect is **half of one standard error** wide.
You can check it yourself with Hanley-McNeil: with A = 0.5 and n1 = n2 = 160, Q1 = A/(2-A) = 0.333, Q2 = 2A²/(1+A) = 0.333, and SE = sqrt([A(1-A) + (n1-1)(Q1-A²) + (n2-1)(Q2-A²)] / n1n2) = sqrt(26.75/25600) = 0.032.
So your result is not evidence that the method fails. It is not evidence of anything. A test that cannot resolve the effect it is looking for returns noise whichever way the noise happens to fall, and that is true of the paper's own out-of-sample too if the author used the same split. Following the paper faithfully was the right instinct; the paper's sample was the weak part.
Two things I would look at before concluding anything about xgboost:
**Where did the 290 early-stopping rows come from?** If they were carved out of the 2,436, fine, though early stopping on 290 rows is itself a decision fitted to a small sample and it eats into your effective degrees of freedom. If they came out of the 319, the out-of-sample is no longer out of sample.
**Compute the cost floor before fitting, not after.** You have a 3.9 bp spread. That is a hard number you knew on day one. The useful question is what edge the model must produce to clear it, expressed in whatever units your signal predicts, and only then whether a 319-row test could ever resolve an edge that size. Nine times out of ten that calculation tells you the experiment cannot answer the question and you save yourself the fit. Your 2.81 bp gross against a 3.9 bp spread was outside the feasible set before any model was trained.
That is a reusable rule and it is worth more than the xgboost lesson. The order matters: cost floor, then power, then model.

One caution on the "invert and rerun" advice, because it is the easiest way there is to manufacture an edge that is not there.
Choosing the sign of your signal by looking at whether AUC landed below 0.5 is fitting one bit of the test set. One bit is small, but it is the same class of error as tuning a threshold on the test set, and it converts "no evidence" into an apparent 53.2 AUC for free. If you do invert, the inverted number is no longer out of sample, and you need fresh data before it means anything.
Before that, the question nobody has asked: **is 46.8 distinguishable from 50 at your sample size?** You have not said how many bars you have, and it decides the whole thread. For a proportion the standard error is about `sqrt(0.25/n)`. Your headline gap is 1.6 points, 48.9 against 47.3. To make that two standard errors you need roughly 3,900 bars, and that is the generous version, because your 47.3 base rate is estimated from the same sample rather than known in advance. Ten-minute bars across the paper's five-month window is on the order of 4,000 bars (39 bars a session, about 105 sessions), so by my rough count you are sitting almost exactly on the boundary where 1.6 points and 0 points are the same measurement. Run it for your real n before concluding anything in either direction, including the pessimistic direction: "historical price features will never yield direction" is also a conclusion, and at that n you have not earned it either.
That is a category I have found genuinely useful to keep: a third verdict between pass and fail, *observed but not demonstrated*. Yours looks like a textbook case of it, and so, frankly, does the paper's 60% with no base rate printed next to it.
Third thing, which the paper's 60% also hides: **sign accuracy is cost-blind**, and on ten-minute bars on a 3x leveraged ETF that is the entire game. When I measured costs on a far slower book, monthly rebalancing of US large caps at about 12 rebalances a year, the total came to 163-203 basis points a year, and 150-190 of that was spread alone rather than commission. To be straight about that number: the spread there is estimated with two published estimators rather than measured, and both run high on liquid names, so treat it as an upper bound. The point stands anyway. At ten-minute bars you cross that toll orders of magnitude more often, so I would want the edge expressed in basis points per trade against one spread crossing long before I cared about the accuracy.
The two numbers worth posting back: your n, and your mean return per bar net of one spread crossing. If the second is negative, the first does not matter.

319 changes the whole picture, and it is better news than "I learned how not to use xgboost".
At an out-of-sample n of 319, split roughly evenly between classes, the standard error on an AUC near 0.5 is about **0.032**. That is 3.2 percentage points. The gap you were measuring is 1.6 points. So the thing you are trying to detect is **half of one standard error** wide.
You can check it yourself with Hanley-McNeil: with A = 0.5 and n1 = n2 = 160, Q1 = A/(2-A) = 0.333, Q2 = 2A²/(1+A) = 0.333, and SE = sqrt([A(1-A) + (n1-1)(Q1-A²) + (n2-1)(Q2-A²)] / n1n2) = sqrt(26.75/25600) = 0.032.
So your result is not evidence that the method fails. It is not evidence of anything. A test that cannot resolve the effect it is looking for returns noise whichever way the noise happens to fall, and that is true of the paper's own out-of-sample too if the author used the same split. Following the paper faithfully was the right instinct; the paper's sample was the weak part.
Two things I would look at before concluding anything about xgboost:
**Where did the 290 early-stopping rows come from?** If they were carved out of the 2,436, fine, though early stopping on 290 rows is itself a decision fitted to a small sample and it eats into your effective degrees of freedom. If they came out of the 319, the out-of-sample is no longer out of sample.
**Compute the cost floor before fitting, not after.** You have a 3.9 bp spread. That is a hard number you knew on day one. The useful question is what edge the model must produce to clear it, expressed in whatever units your signal predicts, and only then whether a 319-row test could ever resolve an edge that size. Nine times out of ten that calculation tells you the experiment cannot answer the question and you save yourself the fit. Your 2.81 bp gross against a 3.9 bp spread was outside the feasible set before any model was trained.
That is a reusable rule and it is worth more than the xgboost lesson. The order matters: cost floor, then power, then model.

One caution on the "invert and rerun" advice, because it is the easiest way there is to manufacture an edge that is not there.
Choosing the sign of your signal by looking at whether AUC landed below 0.5 is fitting one bit of the test set. One bit is small, but it is the same class of error as tuning a threshold on the test set, and it converts "no evidence" into an apparent 53.2 AUC for free. If you do invert, the inverted number is no longer out of sample, and you need fresh data before it means anything.
Before that, the question nobody has asked: **is 46.8 distinguishable from 50 at your sample size?** You have not said how many bars you have, and it decides the whole thread. For a proportion the standard error is about `sqrt(0.25/n)`. Your headline gap is 1.6 points, 48.9 against 47.3. To make that two standard errors you need roughly 3,900 bars, and that is the generous version, because your 47.3 base rate is estimated from the same sample rather than known in advance. Ten-minute bars across the paper's five-month window is on the order of 4,000 bars (39 bars a session, about 105 sessions), so by my rough count you are sitting almost exactly on the boundary where 1.6 points and 0 points are the same measurement. Run it for your real n before concluding anything in either direction, including the pessimistic direction: "historical price features will never yield direction" is also a conclusion, and at that n you have not earned it either.
That is a category I have found genuinely useful to keep: a third verdict between pass and fail, *observed but not demonstrated*. Yours looks like a textbook case of it, and so, frankly, does the paper's 60% with no base rate printed next to it.
Third thing, which the paper's 60% also hides: **sign accuracy is cost-blind**, and on ten-minute bars on a 3x leveraged ETF that is the entire game. When I measured costs on a far slower book, monthly rebalancing of US large caps at about 12 rebalances a year, the total came to 163-203 basis points a year, and 150-190 of that was spread alone rather than commission. To be straight about that number: the spread there is estimated with two published estimators rather than measured, and both run high on liquid names, so treat it as an upper bound. The point stands anyway. At ten-minute bars you cross that toll orders of magnitude more often, so I would want the edge expressed in basis points per trade against one spread crossing long before I cared about the accuracy.
The two numbers worth posting back: your n, and your mean return per bar net of one spread crossing. If the second is negative, the first does not matter.