Defeating DFIELD6 (a direction field solver for Matlab)

I was teaching students about the existence and uniqueness theorems for first order differential equations; what I taught was the following version:

Given and an open rectangle and an initial point
a) If is continuous over then there is at least one solution where and the solution is valid so long as the graph of stays in .
b) if, in addition, is continuous over then the solution is unique.

Ok, ok, I know that there are stronger statements of the uniqueness theorem but this is fine for a first pass.

We then talked about what this means: basically, if the conditions of the uniqueness theorem are met, then solution curves for the same differential equation cannot intersect (if they did, this would be a point where uniqueness does not hold).

So, for grins, I plotted the direction field (using Matlab and dfield6) for and showed how the solver struggled along the axis due to the partial derivative not existing there. But then:

Check out the solutions below the axis. What is going on there? We should have uniqueness!

Then note:
1. These solutions are NOT following the direction fields…
2. The direction field itself is wrong…do you see why? And
3. Note that the “solution” curves have some local maximums and local minimums…that is impossible!

Let’s look at the differential equation again:
. We see the following immediately:
1. is always greater than or equal to zero and can only equal zero when . Hence the lines with “negative” slope in the the direction field are bogus.
2. Any curve can only have relative maximums or minimums when and when . Our solver put in some bogus ones.

We can work on this “by hand” and see that the unique solution meeting is which is valid so long as

We can also examine the differential equation itself to see that the “curves of constant slope” are of the form ; example: along the hyperbolas and we should have slopes equal to 1 in the direction field. Clearly this is not being accurately represented by dfield6 for this differential equation!

So what is going on?

Hint: try to solve this differential equations with Matlab Runge-Kutta methods (and others); what you’ll find is that the software is using complex roots for the “raised to the 4-5’ths” function.

Upshot: we are still smarter than the software.

Classic Overfitting

One common mistake that people sometimes make when they model things is the mistake of overfitting known results to past data.
Life is complicated, and if one wants to find a correlation of outcomes with past conditions, it really isn’t that hard to do.

Here Nate Silver calls out a case of overfitting; in this case someone has a model that is supposed to be able to predict the outcome of a presidential election. It has been “proven” right in the past.

Mr. Silver writes:

If there are, say, 25 keys that could defensibly be included in the model, and you can pick any set of 13 of them, that is a total of 5,200,300 possible combinations. It’s not hard to get a perfect score when you have that large a menu to pick from! Some of those combinations are going to do better than others just by chance alone.

In addition, as I mentioned, at least a couple of variables can credibly be scored in either direction for each election. That gives Mr. Lichtman even more flexibility. It’s less that he has discovered the right set of keys than that he’s a locksmith and can keep minting new keys until he happens to open all 38 doors.

By the way — many of these concerns also apply to models that use solely objective data, like economic variables. These models tell you something, but they are not nearly as accurate as claimed when held up to scrutiny. While you can’t manipulate economic variables — you can’t say that G.D.P. growth was 5 percent when the government said it was 2 percent, at least if anyone is paying attention — you can choose from among dozens of economic variables until you happen to find the ones that pick the lock.

These types of problems, which are technically known as overfitting and data dredging, are among the most important things you ought to learn about in a well-taught econometrics class — but many published economists and political scientists seem to ignore them when it comes to elections forecasting.

In short, be suspicious of results that seem too good to be true. I’m probably in the minority here, but if two interns applied to FiveThirtyEight, and one of them claimed to have a formula that predicted 33 of the last 38 elections correctly, and the other one said they had gotten all 38 right, I’d hire the first one without giving it a second thought — it’s far more likely that she understood the limitations of empirical and statistical analysis.

I’d recommend reading the rest of the article. The point isn’t that the model won’t be right this time; in fact if one goes by the current betting market, there is about a 50 percent chance (slightly higher) that it will be right. But that doesn’t mean that it is useful.