Pairs Trading in Python: Cointegration and Mean Reversion

Every directional strategy has the same hidden bet baked into it: that the market keeps going the way it has been going. Long-only momentum, trend following, even most mean-reversion systems — they all live or die by the overall drift of the market. When 2022 arrives and everything correlated to 1 goes down together, the “diversified” book turns out to have been one trade all along.

Pairs trading in Python is the classic escape hatch. Instead of betting on where the market goes, you bet on the relationship between two securities that historically move together: when the gap between them stretches too far, you bet it snaps back. Done right, the position is market-neutral — it can make money in a bull market, a bear market, or a flat one, because it only cares about the spread between two assets, not the level of either.

The catch is that “two things that move together” is not the same as “two things you can trade against each other.” Correlation is not enough, and trading on correlation alone is one of the most expensive beginner mistakes in statistical arbitrage. The tool that actually matters is cointegration.

By the end of this article you will have:

  • A clear intuition for why cointegration — not correlation — is what makes a pair tradeable.
  • A working statsmodels pipeline that tests a pair and builds a tradeable spread.
  • A market-neutral z-score backtest on real ETF data, with the look-ahead traps named out loud.

1. Why a single-asset strategy is a disguised market bet

Suppose you build a beautiful mean-reversion system on a single stock. It buys dips, sells rips, and the equity curve looks smooth. Then the company’s sector rotates out of favor for eighteen months and the stock grinds down the whole time. Your “mean” was a moving target, and every dip you bought was a falling knife.

The problem is that a single price series has no anchor. There is no law of physics that says a stock has to return to any particular level. Its “fair value” drifts with earnings, rates, sentiment, and a hundred other things you are not modeling.

A pair gives you an anchor. If two companies are in the same business — two soft-drink makers, two oil majors, two country ETFs driven by the same commodities — then whatever macro force pushes one up tends to push the other up too. The difference between them, the spread, has a much better claim to being mean-reverting than either price on its own. When one runs ahead of the other for no fundamental reason, you short the expensive leg, buy the cheap leg, and wait for the gap to close. Your exposure to the market as a whole roughly cancels out.

That cancellation is the whole point. It is also where the rigor has to come in, because not every correlated pair has a stable spread.


2. Cointegration vs correlation (the intuition)

Here is the distinction that trips up most people, with no heavy math.

Correlation measures whether two series move in the same direction day to day. Two random walks can be highly correlated over a window and then wander arbitrarily far apart forever. Correlation says nothing about whether they stay close.

Cointegration is stronger: it says that some linear combination of the two series is stationary — it has a stable mean and reverts to it. The two prices can each wander wherever they like, but they are tied together so that the spread between them keeps coming back.

The standard mental picture is the drunk and her dog. The drunk staggers home on a random walk; her dog wanders on its own random walk. Each path on its own is unpredictable and can go anywhere. But they are joined by a leash. The distance between them is bounded — it stretches and contracts but always pulls back. The two positions are non-stationary; the leash (the spread) is stationary. That leash is cointegration, and it is exactly what you trade.

Crucially, two series can be strongly correlated without being cointegrated (they drift apart for good), and — more rarely — cointegrated without being strongly correlated day to day. For pairs trading, cointegration is the property you need, and there is a formal test for it.


3. Setting up the environment

pip install yfinance statsmodels pandas numpy matplotlib

Imports for the whole article:

import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt
import statsmodels.api as sm
from statsmodels.tsa.stattools import coint, adfuller

4. Getting data and choosing a candidate pair

A good candidate pair should have an economic reason to move together — that is what stops the relationship from being a coincidence that evaporates out of sample. A textbook example is EWA (the iShares Australia ETF) and EWC (iShares Canada). Both economies are commodity-heavy and rate-sensitive, so the two ETFs are pushed around by similar macro forces.

tickers = ["EWA", "EWC"]
prices = yf.download(tickers, start="2010-01-01", end="2025-01-01",
                     auto_adjust=True)["Close"].dropna()

ewa = prices["EWA"]
ewc = prices["EWC"]

Plot them on the same axis and you will see two lines that clearly travel together but are not glued — exactly the profile you want before running any statistical test.

prices.plot(figsize=(14, 6), title="EWA and EWC closing prices")
plt.ylabel("Price (USD)")
plt.show()

Choosing the pair by eye first, test second matters more than it looks. If you skip the economic story and let an algorithm scan thousands of pairs for the lowest p-value, you walk straight into the data-snooping trap covered in section 8.


5. Testing for cointegration

statsmodels ships the Engle-Granger two-step test as a single function, statsmodels.tsa.stattools.coint. The null hypothesis is no cointegration; a small p-value lets you reject it.

score, pvalue, _ = coint(ewa, ewc)
print(f"Engle-Granger cointegration p-value: {pvalue:.4f}")

A p-value below 0.05 is the usual threshold to treat the pair as cointegrated. Do not stop at a single number, though — the test is sensitive to the sample window. A pair that passes on 2010–2025 may fail on 2015–2020. Re-run the test on a few sub-periods before you trust it.

For intuition, it helps to also run an augmented Dickey-Fuller test directly on the spread once you have built it (next section). Engle-Granger is essentially doing that under the hood, but seeing the ADF p-value on the spread you actually trade makes the result concrete.


6. Building the spread and the z-score signal

Cointegration tells you a stationary combination exists; ordinary least squares tells you the hedge ratio that defines it. Regress one leg on the other and the slope is how many units of EWA you hold against one unit of EWC.

X = sm.add_constant(ewa)
ols = sm.OLS(ewc, X).fit()
hedge_ratio = ols.params["EWA"]

spread = ewc - hedge_ratio * ewa

Confirm the spread is stationary with an augmented Dickey-Fuller test — the null here is non-stationary (a unit root), so again you want a small p-value:

adf_stat, adf_p, *_ = adfuller(spread)
print(f"ADF p-value on the spread: {adf_p:.4f}")

Now turn the spread into a tradeable signal. The raw spread has units of dollars and a level that means nothing on its own; what you care about is how stretched it is right now relative to its recent normal. That is a rolling z-score:

window = 30
spread_mean = spread.rolling(window).mean()
spread_std = spread.rolling(window).std()
zscore = (spread - spread_mean) / spread_std

zscore.plot(figsize=(14, 5), title="Spread z-score (EWC vs EWA)")
plt.axhline(2.0, color="r", ls="--")
plt.axhline(-2.0, color="g", ls="--")
plt.axhline(0.0, color="k", ls="-", lw=0.5)
plt.show()

When the z-score spikes above +2, the spread is unusually rich: short it (short EWC, long EWA). When it drops below -2, the spread is unusually cheap: go long it. When it returns toward zero, close the position. Using a rolling mean and standard deviation rather than the full-sample values is deliberate — it keeps the signal backward-looking, which the next sections lean on hard.


7. Backtesting the pairs trading strategy in Python

The trading rules are a clean z-score band: enter at ±2, exit when the spread normalizes back inside ±0.5.

entry, exit = 2.0, 0.5

longs  = zscore < -entry      # spread cheap  -> long the spread
shorts = zscore >  entry      # spread rich   -> short the spread
exits  = zscore.abs() < exit

position = pd.Series(np.nan, index=zscore.index)
position[longs]  = 1
position[shorts] = -1
position[exits]  = 0
position = position.ffill().fillna(0)

Now the part that separates an honest backtest from a marketing chart. The daily profit of holding the dollar-neutral spread is the position (set yesterday) times the change in the spread. The shift(1) is non-negotiable: you can only act on a z-score after the bar that produced it has closed.

spread_ret = spread.diff()
gross = (ewc + hedge_ratio * ewa)          # capital tied up in both legs
strategy_ret = position.shift(1) * spread_ret / gross.shift(1)

equity = (1 + strategy_ret.fillna(0)).cumprod()
equity.plot(figsize=(14, 6),
            title="Market-neutral pairs trading equity curve")
plt.ylabel("Growth of 1 unit")
plt.show()

Dividing the spread PnL by the gross capital of the two legs turns the price-unit profit into a percentage return, so the metrics below are interpretable:

def sharpe(r):
    r = r.dropna()
    return np.sqrt(252) * r.mean() / r.std()

def max_dd(equity):
    peak = equity.cummax()
    return (equity / peak - 1).min()

print("Sharpe :", round(sharpe(strategy_ret), 2))
print("Max DD :", round(max_dd(equity), 3))

What you should expect from a real pair like this: a modest Sharpe, long flat stretches where the spread sits inside the band and you hold nothing, and — the selling point — an equity curve whose shape has very little to do with the S&P 500’s. That low correlation to the broad market is the entire reason to bother. A pairs strategy is not there to beat buy-and-hold on raw return; it is there to add a return stream that does not move with everything else you own.


8. The traps that quietly ruin pairs trades

Pairs trading looks deceptively simple, and the simple version hides several ways to fool yourself.

  • Look-ahead in the hedge ratio. The OLS above is fit on the entire sample, so your 2011 spread was defined using a beta computed with 2024 data. In production you must estimate the hedge ratio on a rolling or expanding window of past data only — see section 9. The same caution is why the z-score uses a rolling window rather than the full-sample mean and standard deviation.
  • Cointegration is not permanent. A pair can be cointegrated for a decade and then decouple — a merger, a regulatory change, a commodity shock, a constituent change in one of the ETFs. Re-test on a rolling window and be ready to retire a pair when the relationship breaks. A blown-up spread that never reverts is the pairs trader’s version of a falling knife.
  • Data snooping when scanning pairs. If you brute-force every pair in a 500-stock universe, that is ~125,000 tests. At a 5% threshold you would expect roughly 6,000 “significant” pairs by pure chance. Picking the lowest p-value out of that pile is overfitting one level up. Either start from an economic hypothesis (as we did) or apply a multiple-testing correction and validate out of sample — the same discipline a walk-forward optimization brings to parameter tuning.
  • Costs and the short leg. A pairs strategy trades both legs and flips often, so commissions roughly double and turnover is high. The short leg also carries borrow costs and can occasionally be hard to borrow. Subtract a realistic round-trip cost — cost * abs(position.diff()) — from the returns and watch how much of the edge survives. Marginal pairs frequently do not survive even 5 basis points.
  • In-sample band tuning. The ±2 / ±0.5 thresholds and the 30-day window are knobs. Tune them on the full history and you are curve-fitting. Choose them a priori or validate them out of sample.

A strategy whose failure modes you can list is one you can manage. A pairs backtest that looks flawless is usually one with the look-ahead left in.


9. Beyond a static hedge ratio

The single biggest weakness above is the fixed, full-sample beta. The relationship between two assets drifts over time, and a static hedge ratio slowly goes stale. Two ways to fix it:

Rolling OLS. Re-estimate the hedge ratio on a trailing window so the spread is always defined by recent history only:

roll_window = 252  # one year
hedge_roll = (ewc.rolling(roll_window)
                 .cov(ewa)
              / ewa.rolling(roll_window).var())
spread_dynamic = ewc - hedge_roll * ewa

This removes the look-ahead and adapts to a drifting relationship, at the cost of a noisier hedge ratio.

Kalman filter. A more elegant approach treats the hedge ratio as a hidden state that evolves smoothly and updates it one observation at a time — no arbitrary window length, and far less jitter than rolling OLS. The pykalman library makes this a few lines; it is a natural follow-up topic in its own right.

For baskets of three or more assets, the Engle-Granger test no longer applies cleanly — reach for the Johansen test (statsmodels.tsa.vector_ar.vecm.coint_johansen), which finds multiple cointegrating relationships at once.


10. Where to go next

A few directions to push this further:

  • Kalman-filter hedge ratios. Replace the static or rolling beta with a Kalman filter for a smoothly time-varying hedge ratio — the standard production upgrade for a pairs book.
  • Half-life of mean reversion. Fit an Ornstein-Uhlenbeck process to the spread to estimate how fast it reverts, then size your z-score window and holding period to match it instead of guessing 30 days.
  • Add a regime filter. Pairs relationships behave differently in calm versus stressed markets. Gate trades on the market state using a regime model — a clean pairing with the HMM market regimes approach.
  • Scale to a universe. Use vectorbt to scan and backtest hundreds of cointegrated pairs quickly, then borrow the falling-knife defenses from the Bollinger Bands and RSI mean-reversion article to manage the ones that decouple.

Conclusion

Pairs trading is the cleanest way to express a view that has nothing to do with where the market is headed. The strategy itself is a few lines of statsmodels — an OLS hedge ratio, a stationary spread, a z-score band. The hard part, and the part that decides whether the thing makes money out of sample, is the discipline around it: testing cointegration honestly, refusing to snoop thousands of pairs, lagging every signal, and accepting that even a good pair can decouple without warning. Get that discipline right and you have something rare in a retail toolkit — a return stream that genuinely zigs when the rest of your book zags.

Mean-Reversion in Python: Bollinger Bands and RSI

Trend-following gets all the attention, but anyone who has run a moving-average strategy through a sideways market knows the other half of the story: in a range, a trend system buys every false breakout and sells every false breakdown until your account looks like it has been through a paper shredder.

A mean-reversion strategy does the opposite. It assumes that after an unusually sharp move, price tends to snap back toward its recent average. In this article we build one in Python using two of the most popular indicators in technical analysis — Bollinger Bands and the RSI — and, more importantly, we backtest it honestly so you can see when it works and when it quietly bleeds.

By the end you will have:

  • A clear, no-nonsense explanation of what Bollinger Bands and the RSI actually measure.
  • A runnable pandas implementation of both, from scratch.
  • A complete mean-reversion backtest on SPY, with a fair comparison to buy & hold.
  • A frank list of the ways this strategy can fool you.

1. Why mean-reversion, and when it works

Markets alternate between two behaviors. In a trending phase, today’s move predicts tomorrow’s: strength begets strength. In a mean-reverting phase, today’s move predicts the opposite: an overextended drop tends to be followed by a bounce.

A mean-reversion strategy is a bet on the second behavior. The logic:

  • Price has dropped sharply below its recent average.
  • We assume the drop is an overreaction.
  • We buy, expecting a snap back to the mean.
  • We sell once price has reverted.

This works beautifully in choppy, range-bound markets — exactly the environment where trend systems struggle. The flip side, which we will return to in the traps section, is that the same strategy is dangerous in a strong downtrend, where every dip looks like a buying opportunity right up until the bottom falls out. Knowing which regime you are in matters, which is why this pairs naturally with regime detection — see the HMM market regimes article.

We need two things: a way to measure “how far is price from its average” (Bollinger Bands) and a confirmation that the move is genuinely stretched (RSI).


2. Bollinger Bands and the RSI in plain English

Bollinger Bands wrap a price chart in a statistical envelope. Three lines:

  • A middle band — a simple moving average, typically 20 days.
  • An upper band — the middle band plus two standard deviations of price.
  • A lower band — the middle band minus two standard deviations.

Because the bands are built from a rolling standard deviation, they widen when volatility rises and contract when it falls. When price touches the lower band, it sits roughly two standard deviations below its recent mean — a statistically unusual place to be. That is our “stretched to the downside” signal.

The RSI (Relative Strength Index) is a momentum oscillator bounded between 0 and 100. It compares the size of recent up-moves to recent down-moves over a lookback window, classically 14 days. Convention:

  • RSI below 30 → “oversold” — selling pressure has been dominant and may be exhausted.
  • RSI above 70 → “overbought”.

Neither indicator is magic, and either one alone produces a lot of false signals. The point of combining them is confirmation: we only act when the price says stretched (lower Bollinger Band) and the momentum says exhausted (low RSI). Two weak signals that agree are stronger than one.


3. Setting up the environment

pip install yfinance pandas numpy matplotlib

Imports:

import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt

4. Getting the data and computing Bollinger Bands

We use the SPY ETF as a proxy for the S&P 500, on daily bars.

data = yf.download("SPY", start="2005-01-01", end="2025-01-01", auto_adjust=True)
data = data[["Close"]].dropna()
data["log_return"] = np.log(data["Close"] / data["Close"].shift(1))

Bollinger Bands are a few lines of pandas:

window = 20
n_std = 2

data["mid_band"] = data["Close"].rolling(window).mean()
rolling_std = data["Close"].rolling(window).std()
data["upper_band"] = data["mid_band"] + n_std * rolling_std
data["lower_band"] = data["mid_band"] - n_std * rolling_std

A useful derived quantity is the %B, which expresses where price sits inside the bands on a 0–1 scale (0 = on the lower band, 1 = on the upper band):

data["pct_b"] = (data["Close"] - data["lower_band"]) / (data["upper_band"] - data["lower_band"])

5. Computing the RSI from scratch

Plenty of libraries ship an RSI, but it is worth implementing once so you know exactly what you are trading. We use Wilder’s smoothing, the original definition, which is an exponential moving average with alpha = 1 / period:

def rsi(series: pd.Series, period: int = 14) -> pd.Series:
    delta = series.diff()
    gain = delta.clip(lower=0)
    loss = -delta.clip(upper=0)
    avg_gain = gain.ewm(alpha=1 / period, adjust=False).mean()
    avg_loss = loss.ewm(alpha=1 / period, adjust=False).mean()
    rs = avg_gain / avg_loss
    return 100 - (100 / (1 + rs))

data["rsi"] = rsi(data["Close"], period=14)
data = data.dropna()

Quick sanity check — the RSI should spend most of its life between 30 and 70, with brief excursions to the extremes:

print(data["rsi"].describe())

If your RSI is pinned near 0 or 100, you have a bug — usually a sign you forgot to separate gains from losses correctly.


6. From indicators to entry and exit rules

Mean-reversion is a stateful strategy: you are either in a position or you are not, and a single day’s data is not enough to know which. So we define an entry condition and a separate exit condition, then walk through time tracking the state.

The rules:

  • Enter long when the close is below the lower Bollinger Band and the RSI is below 30.
  • Exit when the close climbs back to the middle band (the mean has been reached) or the RSI rises above 50.
  • Otherwise, hold whatever position we already have.
entry = (data["Close"] < data["lower_band"]) & (data["rsi"] < 30)
exit_ = (data["Close"] >= data["mid_band"]) | (data["rsi"] > 50)

position = np.zeros(len(data))
in_position = False
for i in range(len(data)):
    if not in_position and entry.iloc[i]:
        in_position = True
    elif in_position and exit_.iloc[i]:
        in_position = False
    position[i] = 1.0 if in_position else 0.0

data["position"] = position

The explicit loop is not the fastest code in the world, but for 5,000 daily bars it runs instantly and it is impossible to misread. Clarity beats cleverness when a subtle bug means losing real money.

One non-negotiable step: shift the position by one bar before computing returns. The decision to be in the market today is made from yesterday’s close, because you cannot trade on a price that has not printed yet.

data["position"] = data["position"].shift(1).fillna(0)

Skip that line and you build look-ahead bias straight into the backtest — the single most common way a mean-reversion strategy looks brilliant on screen and fails live.


7. Backtesting the strategy

With a clean position series, the backtest is one multiplication:

data["strategy_ret"] = data["position"] * data["log_return"]
data["bh_ret"] = data["log_return"]

equity = np.exp(data[["strategy_ret", "bh_ret"]].cumsum())
equity.columns = ["Bollinger+RSI mean-reversion", "Buy & Hold"]

equity.plot(figsize=(14, 6),
            title="Mean-reversion strategy vs Buy & Hold — SPY")
plt.ylabel("Equity (cumulative, log-return based)")
plt.tight_layout()
plt.show()

Mean-reversion strategies are, by construction, out of the market most of the time — they only hold a position during the recovery from an oversold dip. So do not expect the equity curve to track buy & hold. Expect a flatter line that steps up occasionally and, crucially, sidesteps a chunk of the worst drawdowns.


8. Measuring performance honestly

A picture is not a result. We need numbers, and we need the same numbers for buy & hold so the comparison is fair.

def sharpe(r):
    r = r.dropna()
    return np.sqrt(252) * r.mean() / r.std()

def max_drawdown(equity_curve):
    peak = equity_curve.cummax()
    return (equity_curve / peak - 1).min()

def time_in_market(position):
    return position.mean()

strat = data["strategy_ret"]
bh = data["bh_ret"]

print(f"Sharpe strategy   : {sharpe(strat):.2f}")
print(f"Sharpe buy & hold : {sharpe(bh):.2f}")
print(f"Max DD strategy   : {max_drawdown(np.exp(strat.cumsum())):.2%}")
print(f"Max DD buy & hold : {max_drawdown(np.exp(bh.cumsum())):.2%}")
print(f"Time in market    : {time_in_market(data['position']):.1%}")

The pattern you will usually see: the strategy earns a fraction of buy & hold’s total return — because it is invested only a small share of the time — but its drawdowns are far shallower, and the return per day actually invested is high. That last point is what makes mean-reversion useful as a building block: it produces a stream of returns that is largely uncorrelated with a trend system, so the two combine well in a portfolio.

Judge the strategy on risk-adjusted terms and on what it adds to a blend, not on whether it beats the index outright. It will not, and it is not trying to.


9. The traps you must not ignore

Mean-reversion is genuinely useful, but it has sharp edges. Name them or they will find you.

  • It catches falling knives. The strategy buys oversold dips. In a sustained crash — 2008, March 2020 — “oversold” gets more oversold for weeks. Every backtest of a long-only mean-reversion system will show ugly losses in those windows. A regime filter or a hard stop-loss is not optional for live trading.
  • Look-ahead bias. Covered above, and worth repeating: the shift(1) is the difference between a real backtest and a fantasy. Any indicator computed on the same bar you act on must be lagged.
  • Parameter overfitting. The 20-day window, 2 standard deviations, RSI 14, the 30/50 thresholds — every one of those is a knob. Tune them all on the full history and you are curve-fitting. Validate with walk-forward optimization so the parameters are chosen out-of-sample.
  • Transaction costs. This strategy trades fairly often. Add a realistic per-trade cost — subtract cost * abs(data["position"].diff()) from the returns — and re-run. Edges that survive on paper sometimes do not survive 5 basis points of friction.
  • Survivorship bias on single stocks. SPY is an index and reasonably safe. Run the same logic on individual stocks and a name that mean-reverted nicely for years can also simply go to zero — the ultimate failed reversion.

A strategy whose failure modes you can recite is one you can manage. A strategy that “just works” in the backtest is one you do not understand yet.


10. Where to go next

A few directions to push this further:

  • Add a regime filter. Only take mean-reversion trades when a regime model says the market is ranging, not trending down. This directly addresses the falling-knife problem — a natural pairing with the HMM market regimes article.
  • Size by conviction. Instead of a binary 0/1 position, scale exposure with how stretched %B and the RSI are. A deeper dip gets more capital.
  • Add a short side. Mirror the rules — short when price pierces the upper band with RSI above 70. Be warned: shorting an index with a long-term upward drift is a structural headwind.
  • Test the indicators as features. Feed %B and the RSI into a classifier rather than hand-coding thresholds — see the feature selection article for how to do that without drowning in noise.

Conclusion

Bollinger Bands and the RSI are two of the most-Googled indicators in trading, and most of what is written about them is either hand-wavy chart astrology or an equity curve with the look-ahead bias quietly left in. Built carefully in Python and backtested honestly, they form a real, if modest, mean-reversion strategy: not a market-beater on total return, but a low-drawdown, low-correlation return stream that earns its place as a component of a larger book.

Walk-Forward Optimization in Python: The Honest Way to Backtest a Trading Strategy


Every quant has been there. You build a strategy, sweep a few parameters on ten years of data, the equity curve looks beautiful, and then live trading turns it into a sawtooth of disappointment. The backtest wasn’t wrong — it was dishonest. It told you what the best parameters were with the benefit of hindsight. That’s not a strategy. That’s a memory.

The cure has a name: walk-forward optimization. It is boring to implement, slow to run, and the resulting equity curves are uglier — which is precisely why most tutorials skip it.

In this article we will:

  • See, with a concrete example, how an in-sample optimization lies.
  • Build a walk-forward optimizer in pure Python (pandas + numpy, no exotic dependencies).
  • Apply it to a tunable SMA-crossover strategy on SPY.
  • Compare the honest equity curve to the seductive one — and discuss what to do when the gap is wide.

1. The backtest that lied

Take the simplest tunable strategy on earth: a moving-average crossover. Go long when the fast SMA crosses above the slow SMA, flat otherwise. Two parameters: fast and slow.

The naive recipe:

  1. Download 20 years of SPY.
  2. Try every combination of fast in [5, 10, 20, 30] and slow in [50, 100, 150, 200].
  3. Pick the pair with the best Sharpe.
  4. Report that Sharpe as “the strategy’s Sharpe”.

That last step is where the lie lives. You have just searched a 16-cell grid for the cell that fits this specific history best. The Sharpe you report is the maximum of 16 random variables, not the expected performance of the strategy. On unseen data, the same parameters will almost always underperform — sometimes by a wide margin.

The fix is conceptually trivial: never evaluate a strategy on data you used to choose its parameters.


2. Walk-forward in plain English

Walk-forward optimization slices the timeline into a sequence of (train, test) windows that march forward in time:

|===== train 1 =====|= test 1 =|
        |===== train 2 =====|= test 2 =|
                |===== train 3 =====|= test 3 =|
                       ...

In each train window you pick the best parameters by your chosen metric. You then apply those frozen parameters to the next test window — and only the returns from the test windows count toward the final equity curve.

Two common flavors:

  • Rolling window: the train window has fixed length and slides forward. Old data is forgotten.
  • Anchored (expanding) window: the train window grows; only the start is fixed.

Rolling is closer to how a real trader behaves (“forget what worked five years ago, the regime has changed”). Anchored is more statistically efficient when you believe the strategy edge is stable. We’ll use rolling here.


3. Setting up the environment

pip install yfinance pandas numpy matplotlib

Imports:

import itertools
import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt

4. Data and the tunable strategy

data = yf.download("SPY", start="2005-01-01", end="2025-01-01", auto_adjust=True)
data = data[["Close"]].dropna()
data["log_return"] = np.log(data["Close"] / data["Close"].shift(1))
data = data.dropna()

The strategy as a pure function — given parameters and a price series, return the strategy’s daily log-returns:

def sma_crossover_returns(close: pd.Series, log_ret: pd.Series,
                          fast: int, slow: int) -> pd.Series:
    sma_fast = close.rolling(fast).mean()
    sma_slow = close.rolling(slow).mean()
    signal = (sma_fast > sma_slow).astype(int)
    # shift by 1 to use yesterday's signal for today's return → no look-ahead
    return signal.shift(1) * log_ret

And the scoring metric — annualized Sharpe of daily log-returns:

def sharpe(r: pd.Series) -> float:
    r = r.dropna()
    if r.std() == 0 or len(r) < 20:
        return -np.inf
    return np.sqrt(252) * r.mean() / r.std()
&#91;/code&#93;

<hr />

<h2>5. The naive full-sample optimization (the trap)</h2>

<p>For reference, let's compute the in-sample optimum the way most blog posts do it:</p>

[code language="python"]
fast_grid = [5, 10, 20, 30]
slow_grid = [50, 100, 150, 200]
grid = [(f, s) for f, s in itertools.product(fast_grid, slow_grid) if f < s&#93;

scores = {}
for f, s in grid:
    r = sma_crossover_returns(data&#91;"Close"&#93;, data&#91;"log_return"&#93;, f, s)
    scores&#91;(f, s)&#93; = sharpe(r)

best = max(scores, key=scores.get)
print("Best in-sample params:", best, "Sharpe:", scores&#91;best&#93;)
&#91;/code&#93;

<p>You will get a Sharpe somewhere around 0.6–0.8, depending on the seed of history. Remember that number — we will see it shrink.</p>

<hr />

<h2>6. Building the walk-forward loop</h2>

<p>Three knobs:</p>
<ul>
  <li><code>train_years</code>: length of each training window.</li>
  <li><code>test_months</code>: length of each test window — also how often we re-optimize.</li>
  <li>The parameter grid (kept identical to be fair).</li>
</ul>

[code language="python"]
def walk_forward(close: pd.Series, log_ret: pd.Series,
                 grid: list, train_years: int = 5, test_months: int = 6) -> pd.DataFrame:
    train_days = train_years * 252
    test_days = test_months * 21

    records = []
    oos_returns = pd.Series(index=log_ret.index, dtype="float64")

    start = train_days
    while start + test_days <= len(log_ret):
        train_slice = slice(start - train_days, start)
        test_slice = slice(start, start + test_days)

        # optimize on the training window
        best_params, best_score = None, -np.inf
        for params in grid:
            r = sma_crossover_returns(close.iloc&#91;train_slice&#93;,
                                      log_ret.iloc&#91;train_slice&#93;, *params)
            s = sharpe(r)
            if s > best_score:
                best_score, best_params = s, params

        # apply frozen params on the test window
        # Note: we need a small lookback into training so SMAs are warm
        lookback = max(p[1] for p in grid)
        eval_slice = slice(start - lookback, start + test_days)
        r_test = sma_crossover_returns(close.iloc[eval_slice],
                                       log_ret.iloc[eval_slice], *best_params)
        r_test = r_test.iloc[lookback:]  # drop the warm-up portion
        oos_returns.iloc[test_slice] = r_test.values

        records.append({
            "train_end": log_ret.index[start - 1],
            "test_end":  log_ret.index[start + test_days - 1],
            "params":    best_params,
            "is_sharpe": best_score,
        })
        start += test_days

    return oos_returns.dropna(), pd.DataFrame(records)

Two details worth slowing down on:

  1. The warm-up lookback. When you start a new test segment, the slow SMA needs slow past observations to even exist. If you compute SMAs only on the test slice, you throw away the first ~200 trading days of every test window. Including a lookback into the training data fixes this without leaking information — the prediction at day t still only uses prices up to t.
  2. Shift by one. The strategy already shifts the signal by one inside sma_crossover_returns, so today’s position is decided by yesterday’s close. This is non-negotiable. Forget it once and your beautiful walk-forward is just an elaborate look-ahead bias.

7. Running it and stitching the out-of-sample curve

oos_returns, log = walk_forward(data["Close"], data["log_return"], grid,
                                train_years=5, test_months=6)

oos_equity = np.exp(oos_returns.cumsum())
naive_returns = sma_crossover_returns(data["Close"], data["log_return"], *best)
naive_returns = naive_returns.loc[oos_returns.index]
naive_equity = np.exp(naive_returns.cumsum())
bh_equity = np.exp(data["log_return"].loc[oos_returns.index].cumsum())

fig, ax = plt.subplots(figsize=(14, 6))
oos_equity.plot(ax=ax, label="Walk-forward (honest)")
naive_equity.plot(ax=ax, label="In-sample optimum (looks great)")
bh_equity.plot(ax=ax, label="Buy & Hold")
ax.set_title("SMA crossover on SPY — three views of the same strategy")
ax.set_ylabel("Equity (log-return cumulative)")
ax.legend()
plt.tight_layout()
plt.show()

The in-sample curve and the walk-forward curve will almost never agree. On SPY with this grid, the in-sample version reports a Sharpe near 0.7; the walk-forward usually lands between 0.2 and 0.4. That gap is your overfitting tax — it is what you were silently paying every time you took an in-sample backtest at face value.

print("Sharpe walk-forward :", sharpe(oos_returns))
print("Sharpe in-sample    :", sharpe(naive_returns))
print("Sharpe buy & hold   :", sharpe(data["log_return"].loc[oos_returns.index]))

8. Parameter stability — the diagnostic that matters most

A high walk-forward Sharpe means little if the optimizer is jumping wildly between parameter sets in adjacent windows. That’s a sign the “edge” you’re capturing is noise, and next month’s chosen parameters will be the wrong ones.

fig, axes = plt.subplots(2, 1, figsize=(14, 6), sharex=True)
log_plot = log.copy()
log_plot["fast"] = log_plot["params"].apply(lambda p: p[0])
log_plot["slow"] = log_plot["params"].apply(lambda p: p[1])
log_plot.set_index("test_end")[["fast"]].plot(ax=axes[0], marker="o")
log_plot.set_index("test_end")[["slow"]].plot(ax=axes[1], marker="o")
axes[0].set_title("Selected `fast` parameter over time")
axes[1].set_title("Selected `slow` parameter over time")
plt.tight_layout()
plt.show()

What you want to see: long flat stretches with occasional changes. What you don’t want to see: a different pair every window. If the picks look like white noise, the grid is searching too aggressively for the test period length — increase train_years, shrink the grid, or accept that the strategy has no robust edge here.


9. Pitfalls that quietly ruin walk-forwards

Even when you do the basic loop right, a handful of subtler mistakes can re-inject the very bias you were trying to remove.

  • Window-length p-hacking. If you also tune train_years and test_months by looking at the final equity curve, you are back to overfitting — one level up. Decide on these knobs a priori and don’t touch them.
  • No transaction costs. SMA crossovers flip often. A round-trip cost of even 5 bps can knock 30% off the apparent Sharpe. Subtract cost * abs(signal.diff()) from returns and rerun.
  • Survivorship and look-ahead in the data itself. This matters less for SPY but matters a lot for stock universes — only use data that was available as of each rebalance date.
  • Multiple testing. If you try this strategy, then ten others, then pick the one with the best walk-forward Sharpe, you are again selecting the maximum of N. Walk-forward protects against parameter overfit, not against strategy-shopping.
  • Insufficient test data. A 6-month test window contains ~126 daily returns. Sharpe estimated on that is wildly noisy. Stack many such windows before taking the result seriously — and never trust a single segment.

10. Where to go next

A few directions if you want to push this further:

  • Combinatorial Purged Cross-Validation (López de Prado, Advances in Financial Machine Learning). A more rigorous successor to walk-forward that handles overlapping label horizons and gives a distribution of out-of-sample Sharpes instead of a single number.
  • vectorbt has a from_walk_forward helper that runs the same logic ~100× faster on large grids, useful when you move from a 16-cell grid to a 10,000-cell one.
  • Bayesian optimization with scikit-optimize or optuna, instead of grid search, when each backtest is expensive.
  • Block bootstrap of the walk-forward returns to get a confidence interval on the final Sharpe — a 0.4 ± 0.3 reads very differently from a 0.4 ± 0.05.

Conclusion

Walk-forward optimization will not make a bad strategy good. What it will do is stop a bad strategy from looking good, which is more valuable than it sounds: it ends the cycle of building, deploying, and being surprised. The first time you walk-forward a strategy you were proud of and watch its Sharpe halve, it stings. It also saves you from the much more expensive version of that lesson, the one the market teaches with real money.

Trade safe, and remember: a backtest that doesn’t disappoint you is probably lying to you.

Detecting Market Regimes with a Hidden Markov Model in Python (and Adapting Your Strategy Accordingly)

If you have ever backtested a strategy that looked brilliant on one period and fell apart on the next, you have already met the regime problem. Markets are not stationary: a momentum strategy that prints money during a calm bull run can hemorrhage during a choppy bear market. The same indicator, the same parameters, completely different outcomes.

One elegant way to deal with this is to explicitly model the regime the market is in, and let the strategy adapt. In this article we will use a Hidden Markov Model (HMM) in Python to identify market regimes from price data alone, visualize them on a chart, and run a simple backtest that switches behavior depending on the detected regime.

By the end you will have:

  • A clear intuition of what an HMM does (without the dense math).
  • A working hmmlearn pipeline on real market data.
  • A regime-aware backtest you can extend to your own strategies.

1. Why a single model is rarely enough

Imagine fitting a trend-following strategy on the S&P 500 between 2016 and 2019. Steady up-trend, low volatility — the strategy hugs the index and looks great. Now extend the test to 2020 (Covid crash) or 2022 (rate-hike whipsaw). Suddenly the equity curve is a roller-coaster.

The market is not the same animal in every period. Practitioners often describe at least three “moods”:

  • Calm bull: low volatility, positive drift.
  • Volatile bear: high volatility, negative drift.
  • Range / transition: low drift, mixed volatility.

A single set of parameters cannot be optimal in all three. Instead of trying harder on parameter tuning, we can try to classify which mood the market is in and route to the right behavior.

That is exactly what an HMM gives us.


2. The HMM intuition (no heavy math)

A Hidden Markov Model assumes that:

  1. There is a hidden state (the regime) that we cannot directly observe.
  2. We do observe something that depends on the state — in our case, daily returns and volatility.
  3. The hidden state transitions over time according to a Markov chain: the probability of tomorrow’s regime depends only on today’s regime.

In plain English: “The market is in some mood. We don’t see the mood directly, but we see returns that are typical of that mood. Moods tend to persist, but occasionally flip.”

The HMM training step (Baum-Welch / EM) figures out, from the data alone:

  • The statistical signature of each regime (mean and variance of returns).
  • The transition matrix between regimes.
  • The most likely sequence of hidden states (Viterbi).

We don’t have to label anything by hand — it’s unsupervised.


3. Setting up the environment

pip install yfinance hmmlearn pandas numpy matplotlib

Imports:

import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt
from hmmlearn.hmm import GaussianHMM

4. Getting the data and building features

We will use the SPY ETF as a proxy for the S&P 500.

data = yf.download("SPY", start="2005-01-01", end="2025-01-01", auto_adjust=True)
data = data[["Close"]].dropna()

data["log_return"] = np.log(data["Close"] / data["Close"].shift(1))
data["vol_20"] = data["log_return"].rolling(20).std()
data = data.dropna()

We feed the HMM two features per day:

  • The daily log-return (captures direction).
  • The 20-day rolling volatility (captures the “calm vs panic” axis).
features = data[["log_return", "vol_20"]].values

Why two features? With only returns, the model often confuses “small positive” with “small negative”. Adding volatility gives it a second axis to separate quiet trends from noisy chop.


5. Training a 3-state Gaussian HMM

model = GaussianHMM(
    n_components=3,
    covariance_type="full",
    n_iter=1000,
    random_state=42,
)
model.fit(features)

hidden_states = model.predict(features)
data["regime"] = hidden_states

Three states is a sensible default that maps well to the bull / bear / range mental model. You can try 2 or 4, but interpretability quickly degrades beyond 4.

Let’s inspect what the model learned:

for i in range(model.n_components):
    mean_ret, mean_vol = model.means_[i]
    print(f"Regime {i}: mean return = {mean_ret:.5f}, mean vol = {mean_vol:.5f}")

Typical output (your numbers will vary slightly):

Regime 0: mean return =  0.00078, mean vol = 0.00650   -> calm bull
Regime 1: mean return = -0.00120, mean vol = 0.02400   -> volatile bear
Regime 2: mean return =  0.00010, mean vol = 0.01200   -> mid / transition

Important: the HMM does not label its states. Regime 0 is not necessarily “bull”; you have to look at the means and re-map them. A clean way:

order = np.argsort(model.means_[:, 0])  # sort by mean return ascending
labels = {order[0]: "bear", order[1]: "range", order[2]: "bull"}
data["regime_name"] = data["regime"].map(labels)

6. Visualizing the regimes on the price chart

fig, ax = plt.subplots(figsize=(14, 6))
colors = {"bull": "tab:green", "range": "tab:gray", "bear": "tab:red"}

for name, color in colors.items():
    mask = data["regime_name"] == name
    ax.scatter(data.index[mask], data["Close"][mask],
               s=4, color=color, label=name)

ax.set_title("SPY price colored by HMM-detected regime")
ax.set_ylabel("Price")
ax.legend()
plt.tight_layout()
plt.show()

You should see the red dots concentrate around 2008, March 2020, and 2022 — exactly the periods every trader remembers as painful. That is a sanity check that the model is picking up something real, not noise.


7. A simple regime-aware backtest

The simplest possible rule: be long when the regime is bull, in cash otherwise.

data["signal"] = (data["regime_name"] == "bull").astype(int)
data["signal"] = data["signal"].shift(1)  # avoid look-ahead

data["strategy_ret"] = data["signal"] * data["log_return"]
data["bh_ret"] = data["log_return"]

equity = np.exp(data[["strategy_ret", "bh_ret"]].cumsum())

equity.plot(figsize=(14, 6),
            title="Regime-aware long/cash vs Buy & Hold")
plt.ylabel("Equity (log-return cumulative)")
plt.show()

Useful metrics:

def sharpe(r):
    return np.sqrt(252) * r.mean() / r.std()

def max_dd(equity):
    peak = equity.cummax()
    return (equity / peak - 1).min()

print("Sharpe strategy :", sharpe(data["strategy_ret"].dropna()))
print("Sharpe B&H      :", sharpe(data["bh_ret"].dropna()))
print("Max DD strategy :", max_dd(np.exp(data["strategy_ret"].cumsum())))
print("Max DD B&H      :", max_dd(np.exp(data["bh_ret"].cumsum())))

In most runs you will see the regime-aware version give up some upside in raging bull markets, but cut the worst drawdowns by a wide margin — typically halving the max drawdown while keeping a comparable or better Sharpe. That’s the whole point: the goal is not to beat buy & hold on return, it’s to deliver a smoother ride.


8. The trap you must avoid: look-ahead bias

The code above has a subtle but fatal flaw if you copy it into production: we fit the HMM once, on the entire dataset, and then label the past. That means our 2008 regime labels were informed by 2024 data. In a live setting you obviously don’t have that.

The honest version uses a walk-forward fit: re-train the HMM periodically on a rolling window of past data only.

window = 252 * 5      # 5 years
step = 21             # re-fit monthly
preds = pd.Series(index=data.index, dtype="float")

for end in range(window, len(data), step):
    train = features[end - window:end]
    test = features[end:end + step]
    m = GaussianHMM(n_components=3, covariance_type="full",
                    n_iter=200, random_state=42).fit(train)
    preds.iloc[end:end + step] = m.predict(test)

You then need to re-map state indices to bull/range/bear inside each window, since the HMM picks state numbers arbitrarily on each fit. This is the boring-but-essential plumbing that separates a real backtest from a marketing chart.


9. Limitations to keep in mind

  • HMM is classification, not prediction. It tells you which regime you are likely in now, not what tomorrow’s return will be.
  • State count is fragile. Two runs with different random seeds or slightly different data can produce qualitatively different states. Always set a seed and check the means.
  • Gaussian assumption is wrong. Returns have fat tails. GaussianHMM works in practice but underestimates extreme moves; consider a Student-t or GARCH-augmented variant if that matters for you.
  • Lag. The model needs a few days of new data before it confidently flips regimes. You will always switch out of “bull” after a meaningful drawdown has started, not before. That is fine if your goal is risk reduction, less fine if you expect early warning.

10. Where to go next

A few directions if you want to push further:

  • Combine with a momentum signal. Take long-momentum trades only when the regime is “bull” or “range”, flat in “bear”. This often beats the raw momentum strategy on risk-adjusted basis.
  • Markov-switching GARCH. Models the volatility process itself as regime-switching. More principled for risk management.
  • Multi-asset features. Feed VIX, credit spreads, or yield-curve slope alongside SPY returns. The HMM can then pick up macro-driven regimes you would never see from price alone.
  • Bayesian HMM (pomegranate, pymc). Gives you posterior probabilities for each regime instead of a hard label — much nicer for position sizing.

Conclusion

A Hidden Markov Model is one of the cheapest, most interpretable tools you can add to a Python trading toolbox to make your strategies regime-aware. In a few dozen lines of code you go from “one strategy, one market” to “different behavior for different market moods”, and the resulting equity curve is usually a lot easier to live with — even when raw returns are similar.

How to convert a JSON into a HDF5 file

You scraped a bunch of data from a cryptocurrency exchange API into JSON but you figured that it’s taking too much disk space ? Switching to HDF5 will save you some space and make the access very fast, as it’s optimized for I/O operations. The HDF5 format is supported by major tools like Pandas, Numpy and Keras, data integration will be smooth, if you want to do some analysis.

Flattening the JSON

Most of the time JSON data is a giant dictionary with a lot of nested levels, the issue is that HDF5 doesn’t understand that. If we take the below JSON:

json_dict = {'Name':'John', 'Location':{'City':'Los Angeles','State':'CA'}, 'hobbies':['Music', 'Running']}

The result will look like this in a DataFrame:

Nested DataFrame
Nested DataFrame

We need to flatten the JSON to make it look like a classic table:

Flatten DataFrame
Flatten DataFrame

We’re going to use the flatten_json() function (more info here):

def flatten_json(y):
    out = {}
    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x
    flatten(y)
    return out

Loading into a HDF5 file

Now the idea is to load the flattened JSON dictionary into a DataFrame that we’re going to save in a HDF5 file.

I’m assuming that during scraping we appended each record to the JSON, so we have one dictionary per line:

def json_to_hdf(input_file, output_file):
    
    with pd.HDFStore(output_file) as store:
        with open(input_file, "r") as json_file:
            for i, line in enumerate(json_file):
                try:
                    flat_data = flatten_json(ujson.loads(line))
                    df = pd.DataFrame.from_dict([flat_data])
                    store.append('observations', df)
                except:
                    pass

Let’s break this down.

Line 3: we initialize the HDFStore, this is the HDF5 file, it’s handling the file writing and everything.

Lines 4 & 5: we open the file and read it line per line

Line 7: we transform the line into a JSON dictionary and then we flatten it

Line 8: we transform the flatten dictionary into a Pandas DataFrame

Line 9: we append this DataFrame into the HDFStore

Et voilà, you now have your data in a single HDF5 file, ready to be loaded for your statistical analysis or maybe to generate trading signals, remember, it’s optimized for Pandas and Numpy so it’ll be faster than reading from the original JSON file.

Trading with Coinbase Pro (GDAX) API in Python

Coinbase Pro (formerly known as GDAX) is one of the biggest cryptocurrency exchange, you can trade a large panel of cryptocurrencies against USD, EUR and GBP. I chose to trade on Coinbase Pro because it supports a lot of pairs and the liquidity is usually very good, we can easily implement an algorithmic trading strategy on this exchange.

The most traded currencies are:
– Bitcoin (BTC)
– Ethereum (ETH)
– yearn.finance (YFI)
– Litecoin (LTC)

The Setup

Fortunately for us, Coinbase Pro provides an API to get market data, to get balances for each currency and to send buy/sell orders to the market. You can find a documentation here.

I found a Python wrapper for their API on GitHub, this one is super easy to use.
You can install the package like this:

pip install cbpro

Once it’s installed, you need to insert the appropriate import in your code:

import cbpro

Now you need to get an API key in order to be able to retrieve your account balances and to send orders to the market. If you just want to get market data you can skip that part.
Go to https://pro.coinbase.com/profile/api , click on Create new key, now you have the API key and you may need to get some email validation to see the secret key (which you also need). Check the options you want, if you want to trade via the API, just select the appropriate check box, same for withdrawals.

Using the API

In your code, you need to set up the connection so that you can get authenticated:

auth_client = cbpro.AuthenticatedClient(key, b64secret, passphrase)

If you want to get market data for a ticker. Note that authentication is not required for this method:

auth_client.get_product_order_book('BTC-USD')

Now to send an order, it’s pretty simple:

# Buy 0.01 BTC @ 100 USD
auth_client.buy(price='100.00',#USD
size='0.01',#BTC
order_type='limit',
product_id='BTC-USD')

You’ll get a JSON object, with an id for the order that you can track using auth_client.get_fills(order_id=”d0c4560b-4e6d-41d9-e568-48c4bfca13e6″):

{
"id": "d0c4560b-4e6d-41d9-e568-48c4bfca13e6",
"price": "0.10000000",
"size": "0.01000000",
"product_id": "BTC-USD",
"side": "buy",
"stp": "dc",
"type": "limit",
"time_in_force": "GTC",
"post_only": false,
"created_at": "2020-11-20.T10:12:45.12345Z",
"fill_fees": "0.0000000000000000",
"filled_size": "0.00000000",
"executed_value": "0.0000000000000000",
"status": "pending",
"settled": false
}

To manage your risks, you’ll need to retrieve your balances:

balance = auth_client.get_accounts()
print("ETH="+str(balance[0]["balance"]))

With this basic API you can code any algorithmic strategy in Python for Coinbase Pro, you can try to predict the value of a cryptocurrency using our previous tutorials for example.

5 Mistakes To Avoid In Your Trading Strategy

#1 Not learning to code

This one is the most important, before starting anything you should learn about programming. Coding will make you assimilate a certain logic that’s close to mathematical formulas and can help you formalize your trading process. It’s essential to be able to understand everything that’s “under the hood”, what if you strategy starts to slow down after a few months and you’re not able to improve it yourself.

You won’t learn programming in a day, you should take your time to learn and understand the process. Fortunately, there are multiple free methods you can use to learn about Python. You can use websites like EDX, Coursera, and Udacity.

#2 Backtesting and training on the same period

Let’s say you found the perfect strategy that makes +300% in the 2014 period, you may want to backtest it on a different period, the strategy may work in that specific time but it could make you lose a lot on another period. This beginner mistake has a name: overfitting. Ideally you want to split your data set into at least 2 parts: train and test. But if you want to have a rock-solid performance, you can try K-Fold cross validation, it’ll split your data set into K parts, train 1 part and test it on the other ones, and so on.

#3 Not backtesting enough

Backtest, backtest and backtest. Use different time periods, adjust the trading size, the strategy could work by buying 100$ worth of stocks at a time but what if you want to scale it ? You could introduce slippage and of course broker fees.

Backtesting is good but paper trading is better, you should run the strategy in real-time but without any broker connection, this way you can simulate how it’s going to behave with current market situation.

#4 Not having a risk management strategy

Risk management is going to make a difference during bear markets or high-volatility periods. You can limit the maximum exposure and ignore any buying signal if you hit the limit, or automatically close any position older than a few days. These are suggestions, it’s important to make sure you won’t get stuck with a growing loss over time.

#5 Having unreliable data

Your strategy will be based on financial data, either real-time, minute or daily data, a single data point can destroy your profits. You need to make sure it’s coming from a reliable source and not some random websites, a good source is Quandl, some of their datasets are free.

Simple strategy backtesting using Zipline

Zipline is a backtesting engine for Python, if you’re a Quantopian member you should be familiar with it since it’s the one they’re using. It provides metrics about the strategy such as returns, standard deviations, Sharpe ratios etc. basically everything you need to know in order to validate or not a strategy before going live.

Zipline can be install using pip:

pip install zipline

If you’re on Windows I suggest using Conda:

conda install -c Quantopian zipline

Here is the basic structure of a strategy in Zipline:

from zipline.api import order, record, symbol
def initialize(context): pass
def handle_data(context, data): order(symbol('AAPL'), 10) record(AAPL=data.current(symbol('AAPL'), 'price'))

In initialize you can set some global variables used for the strategy such as a list of stocks, certain parameters, the maximum percentage of portfolio invested.
Then handle_data is entered at every tick, that’s where your strategy logic should be. You can check previous articles and incorporate strategies into your code.

Let’s breakdown the handle_data() code.

The order() function let you create an order, here we specify the AAPL ticker (Apple stock) with a quantity of 10. A positive value means you’re buying 10 stocks, a negative value would mean you’re selling the stock.

Then, the record() function allows you to save the value of a variable at each iteration. Here, you’re saving the current stock price under the variable named AAPL, you’ll then be able to retrieve that information in the backtest result, this way you can compare your strategy performance versus the stock price.

Now you want to finally backtest the strategy and see if it’s profitable. To do that, run the following command:

zipline run -f your_strategy.py --start 2015-1-1 --end 2020-1-1 -o your_strategy.pickle

This command is going to run the backtest between 2015-01-01 and 2020-01-01 and output the result into a pickle file for later analysis. The pickle is simply a Pandas DataFrame with a line per day and (a lot of) columns regarding your strategy, such as the return, the number of orders, the portofolio size and so on.

 

Will Bitcoin Ever Be Regulated?

This article by Vlad Andrei was originally published at Albaron Ventures

As Bitcoin and other digital assets continue to grow in adoption and popularity, a common topic for discussion is whether the U.S. government, or any government for that matter, can exert control of its use.

There are two core issues that lay the foundation of the Bitcoin regulation debate:

The digital assets pose a macro-economic risk. Bitcoin and other cryptocurrencies can act as surrogates for an international currency, which throws global economics a curveball. For example, countries such as Russia, China, Venezuela, and Iran have all explored using digital currency to circumvent United States sanctions, which puts the US government at risk of losing its global authority.
Bitcoin logo

International politics and economics are a very delicate issue, and often sanctions are used in place of military boots on the ground, arguably making the world a safer place.

The micro risks enabled by cryptocurrency weigh heavily in aggregate. One of the most attractive features of Bitcoin and other digital assets is that one can send anywhere between a few pennies-worth to billions of dollars of Bitcoin anywhere in the world at any time for a negligible fee (currently around $0.04 to $0.20 depending on the urgency.)

However, in the hands of malicious parties, this could be very dangerous. The illicit activities inherently supported by a global decentralized currency run the gamut: terrorist funding, selling and buying illegal drugs, ordering assassinations, dodging taxes, laundering money, and so on.

Can Bitcoin Even Be Regulated?

Before diving deeper, it’s worth asking whether Bitcoin can be regulated in the first place.

The cryptocurrency was built with the primary purpose of being decentralized and distributed– two very important qualities that could make or break Bitcoin’s regulation.

By being decentralized, Bitcoin doesn’t have a single controlling entity. The control of Bitcoin is shared among several independent entities all over the world, making it nearly impossible for a single entity to wrangle full control over the network and manipulate it as they please.

By being distributed, Bitcoin exists at many different locations at the same time. This makes it very difficult for a single regulatory power to enforce its will across borders. This means that a government or other third party can’t technically raid an office and shut anything down.

That being said, there are several chokepoints that could severely hinder Bitcoin’s adoption and use.

1. Targeting centralized entities: exchanges and wallets

A logical first move is to regulate the fiat onramps (exchanges) , which the United States government has finally been getting around to. In cryptocurrency’s nascent years, cryptocurrency exchanges didn’t require much input or approval from regulatory authorities to run. However, the government started stepping in when cryptocurrency starting hitting the mainstream.

The SEC, FinCEN (Financial Crimes Enforcement Network), and CFTChave all played a role in pushing Know Your Customer (KYC) protocols and Anti-Money Laundering (AML) policies across all exchanges operating within U.S borders.

Cryptocurrency exchanges have no options but to adhere to whatever the U.S. government wants. The vast majority of cryptocurrency users rely on some cryptocurrency exchange to utilize their cryptocurrency, so they will automatically bend to exchange-imposed regulation.

Regulators might not be able to shut down the underlying technology that powers Bitcoin, but they can completely wreck the user experience for the great majority of cryptocurrency users, which serves as enough of an impediment to diminish the use of cryptocurrency for most.

2. Targeting users

The government can also target individual cryptocurrency users. Contrary to popular opinion, Bitcoin (and even some privacy coins) aren’t anonymous. An argument can be made that Bitcoin is even easier to track than fiat because of its public, transparent ledger.

Combined with every cryptocurrency exchange’s willingness to work with U.S. authorities, a federal task force could easily track money sent and received from certain addresses and pinpoint the actual individual with it. Companies such as Elliptic and Chainalysis have already created solid partnerships with law enforcement in many countries to track down illicit cryptocurrency uses and reveals the identities behind the transactions.

Beyond that, we dive into the dark web and more professional illicit cryptocurrency usage. Although trickier, the government likely has enough cyber firepower to snipe out the majority of cryptocurrency-related cybercrime. In fact, coin mixers (cryptoMixer.io), coin swap services (ShapeShift) and P2P bitcoin transactions (localbitcoins.com) have been investigated for several years now and most of them have had to add KYC and adhere to strict AML laws.

Final Thoughts

Ultimately, it’s going to take a lot to enforce any sort of significant global regulation on Bitcoin, with the most important factor being a centralization and consensus of opinion. The majority of the U.S. regulatory alphabet agencies fall into the same camp of “protect the good guys, stop the bad guys”, but there isn’t really a single individual piece of guidance to follow. Currently, cryptocurrencies are regulated in the US by several institutions: CFTC, SEC, IRS, making it difficult to create overarching regulatory guidelines.

In short, yes– Bitcoin can be regulated. In fact, its regulation has already started with the fiat onramps and adherence to strict KYC & AML laws. While in countries such as Ecuador, Bolivia, Egypt and Morocco Bitcoin ownership is illegal, in the US, it would take some bending of the moral fabric of the Constitution in order for cryptocurrency ownership rights to be infringed.

However, it cannot be shut down. There are still ways to buy, sell, and trade Bitcoin P2P, without a centralized exchange. It would take an enormous effort by any government to completely uproot something as decentralized as Bitcoin, but that future seems more dystopian than tangible.

Three Strategies for Choosing What Cryptocurrency to Invest in Next

This article by Steven Buchko was originally published at CoinCentral.com

Finding the Next Bitcoin

You’ve probably thought it at one point or another: “I missed the Bitcoin payday. How do I decide what cryptocurrency to invest in now that I know about the market?”

The bad news: It’s unlikely that any other cryptocurrency will see the same astronomical growth that Bitcoin experienced over the last few years, and impossible to predict it.

The good news: There’s still plenty of opportunities to invest in up-and-coming cryptocurrencies that could potentially bring you 10-100x returns. This comes with a heavy note of caution, because as you may know, cryptocurrencies are incredibly volatile. This is not investment advice, and you should gain/lose money on your own research and intuition.

In this article, we’ll go over some basic strategies you can follow when searching for what cryptocurrency to invest in next. We’re focusing on high risk, high reward options here. If you’re looking for general investment tips, you should check out our article on how to build a proper cryptocurrency portfolio instead.

Scour Initial Coin Offerings (ICOs)

Initial Coin Offerings (ICOs) have quickly become the standard for blockchain startups to raise funding for their project. In an ICO, the team hosts a crowdsale in which you purchase tokens that you can use on their platform. You can also trade these tokens in the secondary market (exchanges) after the ICO.

For example, Golem held an ICO to distribute the first GNT tokens. The purpose of these tokens is to purchase computing power in the Golem network, but traders also buy and sell them on exchanges.

Participating in ICOs can be a lucrative trading strategy. If you invested in the NEO crowdsale (at the time the project was called AntShares), your return on investment (ROI) would be ~160,000% currently. Populous, about 5,000%. OmiseGo, around 4,000%. You get the picture.
ICO ROIs

Source: ICOBench

ICO gains do come with the highest amount of risk, though. The majority of ICOs will fail, and already almost half have done so already.

ICO Research

It’s important that you do your due diligence when picking what cryptocurrency to invest in pre-ICO. There are a ton of things to look at when evaluating a cryptocurrency, but the most important attributes are:

Team and advisors – The team should have experience in blockchain technology or at least the industry that they’re targeting. Preferably both. Having reputable advisors is also a strong sign that the ICO could succeed.
Clear problem/solution – The project’s white paper should clearly define what problem the project is aiming to solve and how the cryptocurrency solves it. Make sure it’s not just a document full of marketing BS.
Token distribution – The team should be distributing over fifty percent of the tokens to crowdsale participants if not much, much more. Be hesitant about projects in which the team and advisors keep a significant proportion of tokens.

Other things to take note of are: any notable partnerships, whether the team has already created a product, and the size of the industry they’re targeting. All of these things could lead to a favorable investment.

Check Lesser Known Exchanges

Even if you missed your chance to participate in an interesting ICO, you can still invest once the coin hits exchanges. At this time, there’s often a brief spike followed by an immediate dump as ICO investors look to cash-in on short-term gains. This is a prime opportunity to get coins you’re interested in for ICO-level (or even lower) prices.

Beyond the short post-ICO period, you still have time to invest in a coin before major exchanges begin to list it. Cryptopia and decentralized exchanges such as IDEX are goldmines for these types of coins. The same research strategies mentioned above apply to coins in this category as well.

IDEX Exchange

Search through coins with a small market cap (<$100 million) that haven’t been listed on a large exchange like Binance yet. You can check CoinMarketCap to see which exchanges coins are on. Make sure you research appropriately and find coins that you believe to have solid fundamentals.

Once you’ve found a coin you’re confident in, purchase it, and (this is the hardest part) wait. It could take days, weeks, or even months for your coin to reach a respectable amount of awareness. If you truly believe in the fundamentals of the coin, though, this timeframe shouldn’t matter. Once the coin joins a major exchange, feel free to trade it accordingly.

Time Important Events

Another popular strategy in selecting what cryptocurrency to invest in is to choose coins based on project roadmaps and event calendars. This is a short-term strategy and usually much harder to execute than the other ones that we’ve covered.

The price of cryptocurrency tends to rise after an important partnership announcement or development milestone. If you follow certain projects on Twitter or are active in their Telegram channel, you usually find out about these announcements ahead of the less involved general public.

With that information, you can sometimes buy into a project early and ride the wave up following the announcement. This has some potential downsides, though. Correct timing is incredibly difficult to accomplish. And, in a bear market, even the most impressive announcements can get crushed under the negative sentiment.

Additionally, the rest of the market may not react to the news the way that you expect. A recent example of this is Verge’s PornHub partnership announcement. While some supporters saw this as positive news, the majority of the market didn’t, and the price crashed accordingly.

Stay Vigilant

Most importantly, you just need to stay vigilant when looking for what cryptocurrency to invest in. New investment opportunities occur every day when you’re actively looking for them. Join subreddits, follow crypto traders on Twitter, constantly research new projects – in essence, engulf yourself in the blockchain space. You never know what gems you’ll stumble upon.