Discussion:
Dan Bernstein's qmail security lessons paper
(too old to reply)
Bill Frantz
2007-12-15 21:26:01 UTC
Permalink
"Some thoughts on security after ten years of qmail 1.0"
Daniel J. Bernstein

<http://cr.yp.to/qmail/qmailsec-20071101.pdf>

This paper has some interesting insights on security engineering a
non-trivial real-world system that should appeal to capability thinkers.
It includes some delightful quotes like:

"I see a huge amount of money and effort being invested in security, and
I have become convinced that most of that money and effort is being
wasted. Most “security” efforts are designed to stop yesterday’s attacks
but fail completely to stop tomorrow’s attacks and are of no use in
building invulnerable software. These efforts are a distraction from
work that does have long-term value."

"I have become convinced that this “principle of least priv-
ilege” is fundamentally wrong. Minimizing privilege might
reduce the damage done by some security holes but almost
never fixes the holes. Minimizing privilege is not the same
as minimizing the amount of trusted code, does not have the
same benefits as minimizing the amount of trusted code, and
does not move us any closer to a secure computer system."

"I have discovered that there are two types of command
interfaces in the world of computing: good interfaces and
user interfaces."

Cheers - Bill

---------------------------------------------------------------------------
Bill Frantz |"We used to quip that "password" is the most common
408-356-8506 | password. Now it's 'password1.' Who said users haven't
www.periwinkle.com | learned anything about security?" -- Bruce Schneier
James A. Donald
2007-12-16 19:29:12 UTC
Permalink
Post by Bill Frantz
"Some thoughts on security after ten years of qmail
1.0" Daniel J. Bernstein
<http://cr.yp.to/qmail/qmailsec-20071101.pdf>
"I have become convinced that this “principle of least
priv- ilege” is fundamentally wrong. Minimizing
privilege might reduce the damage done by some
security holes but almost never fixes the holes.
Minimizing privilege is not the same as minimizing the
amount of trusted code, does not have the same
benefits as minimizing the amount of trusted code, and
does not move us any closer to a secure computer
system."
Obviously minesweeper should not be privileged to read
your mail, record your interactions with bank sites,
mail your passwords to Nigeria, and reformat your hard
drive. Nor should any random word document that
supposedly comes from your friend in email.

But your company web server does need to be pretty much
privileged to run your business, and if a guy from
Nigeria gets control of it, the principle of least
privilege does not help much.

For things like the webserver, what can help a lot is
writing stuff in a language that is immune from buffer
overflows and the like. Buffer overflows are a
particular instance of a more general error: Type
violations - doing things with data that violates type
constraints. Languages such as Java and Lisp are
incapable of that kind of failure, and Lisp's "macros"
encourages one to create types that embody domain
specific correctness, so that one cannot use data
inappropriately for the domain of the data.

Java has dreadful performance problems: Lisp also has
performance problems, but nowhere near as bad as those
of Java, though still bad enough that businesses using
Lisp usually wind up writing key parts of their
applications in C. Still, I notice that truly great
Lisp programmers receive markedly lower pay than truly
great C programmers. Despite the compelling arguments
for using Lisp as an engineering decision, it is an
extremely bad career decision.
Sam Mason
2007-12-16 23:54:13 UTC
Permalink
Post by James A. Donald
Obviously minesweeper should not be privileged to read
your mail, record your interactions with bank sites,
mail your passwords to Nigeria, and reformat your hard
drive. Nor should any random word document that
supposedly comes from your friend in email.
But your company web server does need to be pretty much
privileged to run your business, and if a guy from
Nigeria gets control of it, the principle of least
privilege does not help much.
I fail to see why POLA helps any less when applied server side than on
the desktop. If you run a single, monolithic, web server which handles
everything, then, if I understand things correctly, you've not applied
POLA properly. This single "web server" should be broken down into
smaller components, as you described in the desktop case, and these
components assembled to perform the same functions as the monolithic web
server.

Once the components have been separated, appropriate effort can
be expended on the trusted parts---safe in the knowledge that the
untrusted components can't do anything "bad". This is better than in
the monolithic case, there effort must be universally applied as any
exploitable bugs, anywhere, in the server could cause everything to
fail.


Sam
Sandro Magi
2007-12-17 00:07:35 UTC
Permalink
Post by James A. Donald
But your company web server does need to be pretty much
privileged to run your business, and if a guy from
Nigeria gets control of it, the principle of least
privilege does not help much.
As Sam noted, "the business" is rarely a single monolithic entity, but a
number of smaller processes which are composed to create a final product.

Similarly, "the web server" can be a number of different interacting
servers. Consider a coarse-grained split of resources amongst fairly
standard departments, such as accounting, logistics, sales, human
resources, and so on. Enforceable security property using such
coarse-grained divisions: an attack on your shipping system should not
impact payroll.

Finer-grained divisions can enforce even stronger security properties.

Sandro
David Wagner
2007-12-17 01:12:21 UTC
Permalink
Post by James A. Donald
But your company web server does need to be pretty much
privileged to run your business, and if a guy from
Nigeria gets control of it, the principle of least
privilege does not help much.
I think the principle of least privilege can help even there, if applied
properly. For instance, CGI programs that dynamically generate HTML
content do not need to run as root. It is helpful to run them in a
minimal-privilege jail, so that a single bug in a single CGI program
cannot compromise your entire web site.
Post by James A. Donald
For things like the webserver, what can help a lot is
writing stuff in a language that is immune from buffer
overflows and the like.
That's a good idea, too, but not sufficient on its own (just as the
principle of least privilege is not sufficient on its own).
Well, "dreadful" is rhetoric that appeals to emotion rather than reason.
I think it's a good idea to quantify these claims. How much slower is a
Java-based web server, using metrics that are an appropriate measure of
end-to-end web server performance?

It's my impression that Java is arguably the dominant platform for web
services, in the enterprise world. How do you reconcile that with your
view of Java?
Sandro Magi
2007-12-17 03:07:46 UTC
Permalink
Post by David Wagner
Well, "dreadful" is rhetoric that appeals to emotion rather than reason.
I think it's a good idea to quantify these claims. How much slower is a
Java-based web server, using metrics that are an appropriate measure of
end-to-end web server performance?
It's my impression that Java is arguably the dominant platform for web
services, in the enterprise world. How do you reconcile that with your
view of Java?
The only real performance bottlenecks for garbage collected languages
are tight heaps as found in small devices, and large heaps with virtual
memory where tracing causes thrashing [1]. Unfortunately, server-side
systems can fall into the latter category. Note that this problem is
endemic to ALL tracing garbage collection algorithms, though heuristics
like generations can reduce the thrashing [2].

Additional hardware can make up any performance deficiency, and the
advantages of using Java or ASP.NET with its attendant tooling and lower
risk probably makes up for any additional hardware costs.

Sandro

[1] http://lambda-the-ultimate.org/node/2552
[2] I think a good region-based memory management solution is
"imminent"; now there's a good typed low-level framework [3] which
addresses all the deficiencies of past systems, so only a decent
inference algorithm is missing.
[3] http://lambda-the-ultimate.org/node/2551
Jonathan S. Shapiro
2007-12-17 15:56:58 UTC
Permalink
Post by Sandro Magi
The only real performance bottlenecks for garbage collected languages
are tight heaps as found in small devices, and large heaps with virtual
memory where tracing causes thrashing [1]. Unfortunately, server-side
systems can fall into the latter category.
This is not entirely correct. Or rather, it *should* be correct, but it
is often incorrect in practice.

First, there is the problem of garbage retention. This results from dead
pointer slots on the stack that have not been nullified. Solutions have
long been known, but are not widely implemented.

Second, there is the pragmatic problem that most of the popular
languages of this sort have to interface with C. In many cases (though
perhaps not in Java) this forces them to conservative collectors, which
are horrible.

Finally, there is the problem that when something *does* go wrong with
GC, there are no tools to help you find the culprit (and it isn't clear
how one would design such tools).

Note, however, that all of these costs are associated with GC rather
than safety per se. The only substantive execution-time overheads of
these languages have to do with things like "true" integers, and the
fact that many implementations therefore cannot exploit simplified
register arithmetic. A lot of that can be made to go away by a
sufficiently aggressive compiler, but I do not know if such compiler
technology is widely deployed. I doubt it.

Hotspot is largely irrelevant for server applications. Technologies like
hotspot have a higher startup cost to reach a more efficient stable
state. The problem is that web transactions are short, so you never
really get the payoff from this type of technology.


shap
Sandro Magi
2007-12-17 16:32:35 UTC
Permalink
Post by Jonathan S. Shapiro
First, there is the problem of garbage retention. This results from dead
pointer slots on the stack that have not been nullified. Solutions have
long been known, but are not widely implemented.
I don't see how this is a problem for server-side systems. As you say,
request/response cycles are short, which means the stack is filled then
emptied in a single short request/response. Sometimes a continuation is
built and saved for a subsequent request/response, but the continuation
saves only the relevant state; Waterken behaves like this, but the
continuations are explicit.
Post by Jonathan S. Shapiro
Second, there is the pragmatic problem that most of the popular
languages of this sort have to interface with C. In many cases (though
perhaps not in Java) this forces them to conservative collectors, which
are horrible.
There are relatively few languages that use conservative GC [1]. Mono is
moving to an accurate GC soon. Java, MS .NET, OCaml, SML, Haskell, Ruby,
Python, Perl, various Lisps and Schemes, all use accurate GC. While I'm
not a fan of conservative GC, Hans Boehm has studied the overheads of
and found them to be minimal.
Post by Jonathan S. Shapiro
Finally, there is the problem that when something *does* go wrong with
GC, there are no tools to help you find the culprit (and it isn't clear
how one would design such tools).
Apparently Haskell has been developing a fairly sophisticated heap
profiler. They're in the worst spot resource leak-wise given Haskell's
laziness. I agree that this can be a problem in theory, but I've never
run into it using call-by-value languages.
Post by Jonathan S. Shapiro
Note, however, that all of these costs are associated with GC rather
than safety per se. The only substantive execution-time overheads of
these languages have to do with things like "true" integers, and the
fact that many implementations therefore cannot exploit simplified
register arithmetic. A lot of that can be made to go away by a
sufficiently aggressive compiler, but I do not know if such compiler
technology is widely deployed. I doubt it.
I'm not sure I follow. Just about every language I know of exposes
native integers by default without transparent conversions to big integers.

I would say that the substantial execution overheads result from boxed
representations of simple types like integers and floats, which must
then be traced by the GC. OCaml performs substantial analysis to unbox
such small values.

Escape analysis has also been shown to substantially reduce memory
overheads and execution times. Because .NET retains types at runtime, it
performs such unboxing during JITC for its ValueTypes. Many languages
attempt the same optimizations. Region-based analysis is a
generalization of escape analysis, so they should optimize this memory
overhead even more aggressively.
Post by Jonathan S. Shapiro
Hotspot is largely irrelevant for server applications. Technologies like
hotspot have a higher startup cost to reach a more efficient stable
state. The problem is that web transactions are short, so you never
really get the payoff from this type of technology.
The number of execution paths in a given program are generally fixed, so
Hotspot would optimize these fairly well.

Sandro

[1] http://www.hpl.hp.com/personal/Hans_Boehm/gc/#users
Jonathan S. Shapiro
2007-12-17 16:40:13 UTC
Permalink
Post by Sandro Magi
Post by Jonathan S. Shapiro
First, there is the problem of garbage retention. This results from dead
pointer slots on the stack that have not been nullified. Solutions have
long been known, but are not widely implemented.
I don't see how this is a problem for server-side systems. As you say,
request/response cycles are short, which means the stack is filled then
emptied in a single short request/response.
This is exactly how the problem arises. Typically, the request-response
loop will have some local variable somewhere that gets the "answer" from
all of the processing. This variable must still be in scope at the
response, and additionally there are many local variables that are out
of scope or not reachable from the control flow. Unfortunately those
other local variables usually have non-null values, and the stuff they
point to therefore cannot be GC'd.

The results are very application specific, but the degree of retention
can be very surprising.

Andrew Appel wrote a nice paper on all of this, but I now forget the
title.
Post by Sandro Magi
Post by Jonathan S. Shapiro
Second, there is the pragmatic problem that most of the popular
languages of this sort have to interface with C. In many cases (though
perhaps not in Java) this forces them to conservative collectors, which
are horrible.
There are relatively few languages that use conservative GC [1]. Mono is
moving to an accurate GC soon.
Thank God! This is a lot of what has kept us away from Mono.
Post by Sandro Magi
While I'm
not a fan of conservative GC, Hans Boehm has studied the overheads of
and found them to be minimal.
I have *enormous* respect for Hans, but he has made a number of
inadequately qualified statements in this regard. The cold truth is that
the overheads are extremely application dependent, and no general
conclusion of this sort can be drawn accurately. Our experience with
conservative GC in OpenCM was DISMAL. In fairness, it may also have been
atypical. The point is that nobody (including Hans) can say with
confidence what the cause of the problem was or whether it was typical
or not.
Post by Sandro Magi
Post by Jonathan S. Shapiro
The only substantive execution-time overheads of
these languages have to do with things like "true" integers, and the
fact that many implementations therefore cannot exploit simplified
register arithmetic. A lot of that can be made to go away by a
sufficiently aggressive compiler, but I do not know if such compiler
technology is widely deployed. I doubt it.
I'm not sure I follow. Just about every language I know of exposes
native integers by default without transparent conversions to big integers.
Counter-examples: ML, Lisp, Haskell
Post by Sandro Magi
I would say that the substantial execution overheads result from boxed
representations of simple types like integers and floats, which must
then be traced by the GC. OCaml performs substantial analysis to unbox
such small values.
Yes. Unfortunately this unboxing is not supported by .NET, which is
probably the most widely used runtime implementation of this sort. By
the time you get the JIT layer you've lost a lot of information here.

shap
Sandro Magi
2007-12-17 17:06:33 UTC
Permalink
Post by Jonathan S. Shapiro
This is exactly how the problem arises. Typically, the request-response
loop will have some local variable somewhere that gets the "answer" from
all of the processing. This variable must still be in scope at the
response, and additionally there are many local variables that are out
of scope or not reachable from the control flow. Unfortunately those
other local variables usually have non-null values, and the stuff they
point to therefore cannot be GC'd.
The results are very application specific, but the degree of retention
can be very surprising.
Andrew Appel wrote a nice paper on all of this, but I now forget the
title.
If anyone has a pointer to this, I'd appreciate it! Was it a paper about
web applications or general client/server apps?
Post by Jonathan S. Shapiro
I have *enormous* respect for Hans, but he has made a number of
inadequately qualified statements in this regard. The cold truth is that
the overheads are extremely application dependent, and no general
conclusion of this sort can be drawn accurately. Our experience with
conservative GC in OpenCM was DISMAL. In fairness, it may also have been
atypical. The point is that nobody (including Hans) can say with
confidence what the cause of the problem was or whether it was typical
or not.
The suitability of GC in general is somewhat application-specific, so I
can certainly accept that.
Post by Jonathan S. Shapiro
Post by Sandro Magi
I'm not sure I follow. Just about every language I know of exposes
native integers by default without transparent conversions to big integers.
Counter-examples: ML, Lisp, Haskell
"int" is not transparent in OCaml or in SML; you have to explicitly open
and use the BigInteger or IntInf modules, respectively. In Haskell, you
use Integer for arbitrary precision, or Int for platform integers. You
only pay for what you use. Lisp falls into that "just about every
language" qualifier, along with E. ;-)
Post by Jonathan S. Shapiro
Post by Sandro Magi
I would say that the substantial execution overheads result from boxed
representations of simple types like integers and floats, which must
then be traced by the GC. OCaml performs substantial analysis to unbox
such small values.
Yes. Unfortunately this unboxing is not supported by .NET, which is
probably the most widely used runtime implementation of this sort. By
the time you get the JIT layer you've lost a lot of information here.
.NET carries full type information all the way through to the JIT, and
it has a simple compilation model: if ValueType then stack allocate else
heap allocate. What other information are you referring to?

All the register allocator needs is size information to assign a value
to a register, and it would seem that the JIT would have this information.

Sandro
Mark Miller
2007-12-17 17:40:13 UTC
Permalink
Post by Sandro Magi
I'm not sure I follow. Just about every language I know of exposes
native integers by default without transparent conversions to big integers.
Whatever those native things might be, they are not integers.
--
Text by me above is hereby placed in the public domain

Cheers,
--MarkM
Sandro Magi
2007-12-17 17:51:11 UTC
Permalink
Post by Mark Miller
Post by Sandro Magi
I'm not sure I follow. Just about every language I know of exposes
native integers by default without transparent conversions to big integers.
Whatever those native things might be, they are not integers.
Modular arithmetic has the same semantics as native integers. So they
are integers, from a certain point of view. ;-)

Sandro
Jonathan S. Shapiro
2007-12-17 18:33:58 UTC
Permalink
Post by Sandro Magi
Modular arithmetic has the same semantics as native integers. So they
are integers, from a certain point of view.
I believe you mean to be saying that conventional register-based
fixed-point arithmetic is defined w.r.t. a ring algebra. I agree.

This is precisely why fixed-point arithmetic is NOT integer arithmetic.
Integer arithmetic is defined over an infinite field.
Sandro Magi
2007-12-17 20:03:19 UTC
Permalink
Post by Jonathan S. Shapiro
Post by Sandro Magi
Modular arithmetic has the same semantics as native integers. So they
are integers, from a certain point of view.
I believe you mean to be saying that conventional register-based
fixed-point arithmetic is defined w.r.t. a ring algebra. I agree.
This is precisely why fixed-point arithmetic is NOT integer arithmetic.
Integer arithmetic is defined over an infinite field.
Integers are also defined by a ring algebra [2], not a field [3].

Computers cannot represent true integer arithmetic because of the
infinity. ALL supposed computable "integer arithmetic" is modular
arithmetic [1]. Addition is partial function for both arbitrary
precision integers and native integers, where addition for true integers
is total. I see no significant semantic difference between overflow and
"resource exhaustion".

As Zooko mentioned, this might be a useful psychology debate for the
usability of one particular abstraction over another, but I see little
formal difference.

Sandro

[1] http://mathworld.wolfram.com/ModularArithmetic.html
[2] http://mathworld.wolfram.com/Integer.html
[3] http://mathworld.wolfram.com/Field.html
Jonathan S. Shapiro
2007-12-17 21:05:05 UTC
Permalink
Post by Sandro Magi
Integers are also defined by a ring algebra [2], not a field [3].
Thanks. I stand corrected.
Post by Sandro Magi
Computers cannot represent true integer arithmetic because of the
infinity. ALL supposed computable "integer arithmetic" is modular
arithmetic [1].
Nah. I'ld agree that all *correct* "integer arithmetic" is modular
because of memory limitations, but I have yet to see a BigNum
implementation in the wild that actually checks for this error. :-)
Sandro Magi
2007-12-17 21:13:37 UTC
Permalink
Post by Jonathan S. Shapiro
Post by Sandro Magi
Computers cannot represent true integer arithmetic because of the
infinity. ALL supposed computable "integer arithmetic" is modular
arithmetic [1].
Nah. I'ld agree that all *correct* "integer arithmetic" is modular
because of memory limitations, but I have yet to see a BigNum
implementation in the wild that actually checks for this error. :-)
Well, most programs in the wild don't check for overflow either, they
just throw an exception or terminate; the similarities are striking! ;-)

Sandro
zooko
2007-12-17 22:30:43 UTC
Permalink
Post by Jonathan S. Shapiro
Nah. I'ld agree that all *correct* "integer arithmetic" is modular
because of memory limitations, but I have yet to see a BigNum
implementation in the wild that actually checks for this error. :-)
I just read (most of) "Practical Cryptography" by Ferguson & Schneier
(see my glowing review on the tahoe-dev list [1]), and they describe
a cool little hack to automatically detect bugs in your bignum
package which goes under the cool little name of "wooping". It is
described first in Bos Jurjen's thesis [2, chapter 6], and also in
"Practical Cryptography", which I recommend.

Regards,

Zooko

[1] http://allmydata.org/pipermail/tahoe-dev/2007-December/000271.html
[2] http://citeseer.ist.psu.edu/336144.html
Tony Finch
2007-12-17 16:14:49 UTC
Permalink
Post by Sandro Magi
The only real performance bottlenecks for garbage collected languages
are tight heaps as found in small devices, and large heaps with virtual
memory where tracing causes thrashing.
The other problem is concurrency. Java assumes lots of cross-thread
sharing, so the standard library tends to be rather heavily locked
(causing lots of synchronization overhead), and the GC has to be rather
more complicated than Erlang's per-process heaps or E's vats.

Tony.
--
f.a.n.finch <***@dotat.at> http://dotat.at/
VIKING NORTH UTSIRE: SOUTHERLY 3 OR 4, OCCASIONALLY 5 AT FIRST IN NORTH
VIKING, BECOMING VARIABLE 3 IN NORTH UTSIRE. MODERATE OR ROUGH. FAIR. GOOD.
Sandro Magi
2007-12-17 16:38:12 UTC
Permalink
Post by Tony Finch
The other problem is concurrency. Java assumes lots of cross-thread
sharing, so the standard library tends to be rather heavily locked
(causing lots of synchronization overhead), and the GC has to be rather
more complicated than Erlang's per-process heaps or E's vats.
Indeed. The various escape analyses I referred to in another message do
double-duty to remove locking; if an object can be stack-allocated,
locking and unlocking is pointless after all.

Concurrent GCs as in .NET and Java are indeed more complicated as they
must synchronize with the mutator, though I wouldn't go so far as to say
that Erlang's is simple: their per-process GC is a simple copying
collector, but they use a global shared heap for exchanging messages
which is collected incrementally; generally, maintaining two different
GCs is more challenging than a single GC.

Sandro
James A. Donald
2007-12-17 03:38:26 UTC
Permalink
Post by David Wagner
Well, "dreadful" is rhetoric that appeals to emotion
rather than reason. I think it's a good idea to
quantify these claims. How much slower is a
Java-based web server, using metrics that are an
appropriate measure of end-to-end web server
performance?
My observation that Java has dreadful performance is
based entirely on client side programs.
Post by David Wagner
It's my impression that Java is arguably the dominant
platform for web services, in the enterprise world.
How do you reconcile that with your view of Java?
The choice in practice is between PHP, Perl, Java and
C++. If you write in C++, you will get memory leaks,
and your server will grind to a halt. Java's
performance is not too bad when compared to PHP. Perl
is write only, which means that the if the engineer
wanders off, the businessman is hosed. Businessmen
therefore never specify Perl, but frequently discover
that they are running Perl scripts, long after the fact.

Lisp has no memory leaks, and substantially better
performance than Java, but Lisp programmers are not
interchangeable and readily replaceable. Businessmen
are worried that if they create their website in Lisp,
they yield too much control to the engineer.
Ivan Krstić
2007-12-17 06:47:54 UTC
Permalink
Post by James A. Donald
The choice in practice is between PHP, Perl, Java and
C++.
And increasingly Python, which is the second most widely-used language
at Google, for example.
Post by James A. Donald
Lisp has no memory leaks, and substantially better
performance than Java
This sentence doesn't mean anything. Lisp is a language with many
different widely-used implementations; Java, while having multiple
implementations, tends to be used almost exclusively to refer to Sun's
reference implementation.

Certain Lisp implementations are quite fast at certain tasks. On the
other hand, Java JIT compilers have been getting incredibly good at
optimizing code, and can get uncomfortably close to C performance.

Here's Java 6, HotSpot JVM in server mode, going up against gcc,
getting within 10% of C speed for some tasks:
<http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=java&lang2=gcc
Here's the same JVM doing better than Steel Bank Common Lisp on almost
every test:
<http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=sbcl&lang2=java
Your mileage -- especially with commercial Lisps like Allegro -- may
vary. But in general, beware of handwaving about performance. When not
backed by hard data, it simply looks silly.

--
Ivan Krstić <***@solarsail.hcs.harvard.edu> | http://radian.org
James A. Donald
2007-12-17 21:05:46 UTC
Permalink
Post by Ivan Krstić
Certain Lisp implementations are quite fast at certain
tasks. On the other hand, Java JIT compilers have been
getting incredibly good at optimizing code, and can
get uncomfortably close to C performance.
Here's Java 6, HotSpot JVM in server mode, going up
against gcc, getting within 10% of C speed for some
<http://shootout.alioth.debian.org/gp4/benchmark.php?t
est=all&lang=java&lang2=gcc>
Any language is likely to get within ten percent of any
other on *some* tasks. The Sieve of Eratosthenes used
to be a widely used benchmark, until a number of
compilers incorporated sieve detectors and upon
detecting the sieve, spat out a hand optimized assembly
language sieve.

The test you cite fails to measure the bad Java startup,
which makes java notoriously aggravating in client
programs, though it does not affect server programs
much.

The Java programs usually took up twenty to thirty times
as much memory as C++. In practice, if you are running
C programs, most of your memory is in use, and if you
are running java programs, most of your memory is in
use, which means that when you are running java ...

The extremely poor performance of java programs I
experience is probably due to memory exhaustion on a two
gigabyte machine, plus of course Java's notoriously poor
start up time.
Post by Ivan Krstić
Here's the same JVM doing better than Steel Bank
<http://shootout.alioth.debian.org/gp4/benchmark.php?t
est=all&lang=sbcl&lang2=java>
My comparison of Java and Lisp seems to have been
rendered out of date by the recent improvements in
memory management. I have no recent experience
comparing them. I have recent experience comparing
Java and C++, and these benchmarks are consistent with
my subjective experience: that Java really sucks. But
it does seem, that these days, it sucks less than it
used to and now slightly less than lisp, contrary to
what I claimed.
Ivan Krstić
2007-12-17 21:43:27 UTC
Permalink
Post by James A. Donald
The test you cite fails to measure the bad Java startup,
It doesn't; it's listed as the 'startup' test, where server-mode
Hotspot is 92 times slower to print 'hello world' than C. (It would be
faster in client mode.)
Post by James A. Donald
The Java programs usually took up twenty to thirty times
as much memory as C++.
The linked test agrees with this; Hotspot used between 1.4x to 29x
times more memory than C, mean of 16.27x.
Post by James A. Donald
my subjective experience: that Java really sucks.
My subjective experience agrees entirely, but the pandemic of
performance hand-waving in our industry still grates on me. Anyone's
free to say "I dislike x because of y". When they say "x sucks because
it's slow" (especially if they omit the "when doing y" fragment!), I
want to see numbers, and they better understand basic statistics and
experiment design. Then there's also the stupidity of how
"performance" has come to mean "as fast as possible" for no good
reason[0]. Anyway, this is off-topic for the list, so I'll leave it
here. Cheers,


[0] http://zestyping.livejournal.com/193260.html

--
Ivan Krstić <***@solarsail.hcs.harvard.edu> | http://radian.org
zooko
2007-12-17 14:19:15 UTC
Permalink
Post by David Wagner
Well, "dreadful" is rhetoric that appeals to emotion rather than reason.
I think it's a good idea to quantify these claims. How much slower is a
Java-based web server, using metrics that are an appropriate
measure of
end-to-end web server performance?
It's my impression that Java is arguably the dominant platform for web
services, in the enterprise world. How do you reconcile that with your
view of Java?
This may sound flippant, but I write with consideration and relevant
experience:

I reconcile it by believing that the dominant platform for web
services in the enterprise world is a technically inferior platform.

One shouldn't think that market dominance implies technical
superiority. Neither, of course, should one assume that technical
superiority is the most important criterion for your decisions. If
you want to contribute to the development of the world's enterprise
web services platform, then by all means you should use Java to do so.

It would be really good if we could quantify claims such as "Tool X
is technically superior for task Y.". Presumably we could do that
using the decades-old practices of controlled trials, such as are
used in psychology, human factors engineering, and so on.
Unfortunately such an experiment would be expensive, and few people
have attempted to run one. Here is the best example that I have found:

"An empirical comparison of C, C++, Java, Perl, Python, Rexx, and Tcl
for a search/string-processing program" (2000) by Lutz Prechelt
http://citeseer.ist.psu.edu/311372.html

It is unfortunate that designers of programming languages often
consider themselves to be working in the fields of mathematics or
computer science, and they use the practices and tools of those arts,
when we would be better served if they thought of themselves as
working in psychology and used the standards of measurement required
for publishing results in *that* field.

Regards,

Zooko

P.S. Why is Python so easy to learn and to use? Perhaps because its
immediate ancestor ABC went through many iterations of semi-
controlled, observed user testing during its design.
Bill Frantz
2007-12-17 03:08:25 UTC
Permalink
Post by David Wagner
Post by James A. Donald
But your company web server does need to be pretty much
privileged to run your business, and if a guy from
Nigeria gets control of it, the principle of least
privilege does not help much.
I think the principle of least privilege can help even there, if applied
properly. For instance, CGI programs that dynamically generate HTML
content do not need to run as root. It is helpful to run them in a
minimal-privilege jail, so that a single bug in a single CGI program
cannot compromise your entire web site.
What Bernstein suggests is using tools like language interpreters to
run parts of systems in environments where they can do no harm. In
his paper, he suggests that the parser which parses the "From:"
header and returns a string which is the ID of the sender be run so
that, even if it is completely compromised, the only thing it can do
is return the ID of the sender, and points out that it would
probably be easier to set that ID with a valid header. He mentions
using dataflow analysis when decomposing a system in this manner.

I should add, reading between the lines of this paper, he seems to
assume that privileges can only be controlled at the operating
system level. He doesn't mention capability systems or languages.
The closest he gets to system to confine programs are these two
references:

Ian Goldberg, David Wagner, Randi Thomas, Eric
Brewer, A secure environment for untrusted helper
applications (confining the wily hacker), 6th USENIX
Security Symposium (1996). URL: <http://www.usenix.org/publications/library/proceedings/sec96/goldberg.html>.

David S. Peterson, Matt Bishop, Ra ju Pandey, A
flexible containment mechanism for executing
untrusted code, 11th USENIX Security Symposium
(2002). URL: <http://www.usenix.org/events/sec02/peterson.html>.
Post by David Wagner
Post by James A. Donald
For things like the webserver, what can help a lot is
writing stuff in a language that is immune from buffer
overflows and the like.
That's a good idea, too, but not sufficient on its own (just as the
principle of least privilege is not sufficient on its own).
Yes, SQL injection attacks aren't based on taking over the web
server. They depend on it failing at the devilishly hard problem of
quoting an arbitrary string so some j-random SQL parser will treat
it as data and not instructions.

Cheers - Bill

-------------------------------------------------------------------------
Bill Frantz | The first thing you need when | Periwinkle
(408)356-8506 | using a perimeter defense is a | 16345 Englewood Ave
www.pwpconsult.com | perimeter. | Los Gatos, CA 95032
James A. Donald
2007-12-17 09:31:41 UTC
Permalink
Post by Bill Frantz
Post by James A. Donald
For things like the webserver, what can help a lot
is writing stuff in a language that is immune from
buffer overflows and the like.
Yes, SQL injection attacks aren't based on taking over
the web server. They depend on it failing at the
devilishly hard problem of quoting an arbitrary string
so some j-random SQL parser will treat it as data and
not instructions.
Which is not possible in general. To safely escape user
supplied strings, one needs to use an escape routine
supplied by the people who supply the parser. If one
uses ones own custom escape routine, or an escape
routine that was written for a different database using
a different parser ....

This is a major and basic flaw in SQL, a reflection of a
more basic problem - that interactions between separate
threads on possibly separate machines must ultimately
take the form of a data stream, and we don't have any
good standard way of structuring a stream into calls and
arguments, or rather far too many good standard ways.
David Wagner
2007-12-17 04:13:03 UTC
Permalink
Post by James A. Donald
Lisp has no memory leaks, and substantially better
performance than Java,
Can you back up that statement with quantitative experimental data and an
explanation of the measurement methodology you used? It is easy to claim
that X is better than Y, but if you cannot measure *how much* better X is,
that's often a hint that you don't really know -- you're just guessing.
I do not find such claims credible in the absence of careful measurements.

Do you have experimental measurements to justify and quantify the
claim that Java performance is "dreadful", or are you going on personal
intuition?



"I often say that when you can measure what you are speaking about, and
express it in numbers, you know something about it; but when you cannot
measure it, when you cannot express it in numbers, your knowledge is of
a meagre and unsatisfactory kind; it may be the beginning of knowledge,
but you have scarcely in your thoughts advanced to the state of science,
whatever the matter may be." -- Lord Kelvin
Mark Miller
2007-12-17 15:25:47 UTC
Permalink
Post by David Wagner
"I often say that when you can measure what you are speaking about, and
express it in numbers, you know something about it; but when you cannot
measure it, when you cannot express it in numbers, your knowledge is of
a meagre and unsatisfactory kind; it may be the beginning of knowledge,
but you have scarcely in your thoughts advanced to the state of science,
whatever the matter may be." -- Lord Kelvin
When the topic at issue is inherently quantifiable and measurable, as
it here, that quote is not obviously terrible. But in general, it is
terrible. The stance it supports has done more damage to science and
the growth of human knowledge than just about anything else since the
dark ages. It is also internally contradictory: How could one even
support the claim it makes by expressing it in numbers.

http://www.archive.org/details/counterrevolutio030197mbp
http://ideas.repec.org/a/aea/jeclit/v21y1983i2p481-517.html
http://www.oopsla.org/oopsla2007/index.php?page=sub/&id=265
--
Text by me above is hereby placed in the public domain

Cheers,
--MarkM
Rob Meijer
2007-12-17 13:22:11 UTC
Permalink
Post by James A. Donald
Post by David Wagner
It's my impression that Java is arguably the dominant
platform for web services, in the enterprise world.
How do you reconcile that with your view of Java?
The choice in practice is between PHP, Perl, Java and
C++. If you write in C++, you will get memory leaks,
and your server will grind to a halt.
This is a prety strong statement. There are tools like valgrind
that are very helpfull in eliminating memory leaks, and there
are smart pointer types in both stl and boost that help prevent
many of the sources of memory leaks with C++.
Further the idea that Java programs don't have memory leaks is
something that is based on an oversimplification, Java programs
in practice can leak memory, and if they do they are often harder
to locate than C++ memory leaks.
There are multiple reasons why C++ is less suitable for many web based
applications, but IMO memory leaks should not be considdered as such.

Rob
David Wagner
2007-12-17 17:08:04 UTC
Permalink
Post by zooko
I reconcile it by believing that the dominant platform for web
services in the enterprise world is a technically inferior platform.
One shouldn't think that market dominance implies technical
superiority.
Sure. So one could potentially describe Java's performance
as "inferior but acceptable". That lends a different impression
than "dreadful", doesn't it? My point is that it's better to use
hard numbers, not emotion-laden adjectives, to describe what is
essentially a quantitative phenomenom. The choice of adjectives
is too easily influenced by personal preferences.
Post by zooko
It would be really good if we could quantify claims such as "Tool X
is technically superior for task Y.". Presumably we could do that
using the decades-old practices of controlled trials, such as are
used in psychology, human factors engineering, and so on.
Unfortunately such an experiment would be expensive, and few people
"An empirical comparison of C, C++, Java, Perl, Python, Rexx, and Tcl
for a search/string-processing program" (2000) by Lutz Prechelt
http://citeseer.ist.psu.edu/311372.html
It is unfortunate that designers of programming languages often
consider themselves to be working in the fields of mathematics or
computer science, and they use the practices and tools of those arts,
when we would be better served if they thought of themselves as
working in psychology and used the standards of measurement required
for publishing results in *that* field.
Interesting point!
David Wagner
2007-12-17 23:18:24 UTC
Permalink
Post by Sandro Magi
Computers cannot represent true integer arithmetic because of the
infinity. ALL supposed computable "integer arithmetic" is modular
arithmetic [1].
That doesn't sound right to me. Generally bignum libraries support
"integer arithmetic". The larger the integer you try to operate on,
the greater the time and space it takes to complete the operation.
Some operations on large integers may not complete in a timely manner,
or may not complete successfully at all (e.g., because they threw a
OutOfMemory exception). But that's different in an important way from
saying that they use modular arithmetic. If the bignum computation
completes, then you know that it was performed correctly in the integers
(not in some ring of integers mod 2^n). That's an important property
in some applications. In some applications, reducing mod 2^n leads
to an incorrect answer, while failing to complete in a timely manner
leads to no answer at all. If no answer at all is better than an
incorrect answer (if it is preferable to fail safe), then bignum
arithmetic may be preferable over modular arithmetic.
Sandro Magi
2007-12-18 00:01:48 UTC
Permalink
Post by David Wagner
That doesn't sound right to me. Generally bignum libraries support
"integer arithmetic". The larger the integer you try to operate on,
the greater the time and space it takes to complete the operation.
Some operations on large integers may not complete in a timely manner,
or may not complete successfully at all (e.g., because they threw a
OutOfMemory exception). But that's different in an important way from
saying that they use modular arithmetic.
I say there's no semantic difference at all. Consider the signature of
addition for register-sized integers:

val (+): int -> int -> int

This is really a partial function, since it could overflow. We can make
it total as follows:

type Result = Overflow | Success of int
val (+): int -> int -> Result

Consider now BigInt addition:

val (+): BigInt -> BigInt -> BigInt

But this too is a partial function on any realizable machine, as we
could exhaust memory trying to compute it. We can make this addition
function total as follows:

type Result = OutOfMemory | Success of BigInt
val (+): BigInt -> BigInt -> Result

There is an striking symmetry here, and I say that semantically, the
signature of addition is really the same:

type Result = ResourcesExhausted | Success of int
val (+): int -> int -> Result

Overflow is a type of resource exhaustion, or contrarily, memory limits
place an overflow bound on BigInt. Any computable integer conforms to
this semantics. True arbitrary precision integers can only be used in
symbolic computations where they are not realized.

Further, that addition signature is total, so clients *must* deal with
overflow/resource exhaustion, regardless of the underlying implementation.

Sandro
David Wagner
2007-12-18 01:15:31 UTC
Permalink
ALL supposed computable "integer arithmetic" is modular arithmetic [1].
That doesn't sound right to me.
I say there's no semantic difference at all. Consider the signature of
val (+): int -> int -> int
This is really a partial function, since it could overflow. We can make
type Result = Overflow | Success of int
val (+): int -> int -> Result
That's not modular arithmetic. Modular arithmetic would be

val modplus: int -> int -> int

where modplus x y returns x+y mod 2^w, for some word size w.

I accept the similarities between "fixed-precision integer arithmetic
with exception on overflow" and "unlimited-precision integer arithmetic
with exception on resource exhaustion", but neither of those are
modular arithmetic.
Sandro Magi
2007-12-18 04:20:12 UTC
Permalink
Post by David Wagner
That's not modular arithmetic. Modular arithmetic would be
val modplus: int -> int -> int
where modplus x y returns x+y mod 2^w, for some word size w.
I accept the similarities between "fixed-precision integer arithmetic
with exception on overflow" and "unlimited-precision integer arithmetic
with exception on resource exhaustion", but neither of those are
modular arithmetic.
Sorry, I got my signals crossed. You are correct. When I made the
statement, I was considering the extreme, where the largest integer one
could conceivably operate on is limited by addressable memory, and thus
has modular semantics. For instance, keep an operand in a register and
destructively update all of memory holding the large integer.

The addition function I outlined does not wrap, but it does have a more
usable semantics. BigInt addition can be made to wrap, but the semantics
are undesirable. For instance, if only w bits are available, then
addition would wrap on 2^w. Yuck. Or allow more explicit resource
management with effectful semantics: clients preallocate an integer with
word size w and with the possibility of failure, then fill:

type Result = Exhausted | Success of PreInt
val new_sized : int -> Result
val (+) : int -> int -> PreInt -> int

You would need a linear type system to ensure that the PreInt cannot be
used after the addition. Actually, using linear types I think a more
usable semantics is possible, but you get the idea.

I'd be interested in more pleasant alternatives, as I think totality is
critical for writing correct software; the more total functions over
partial functions used, the fewer the failure modes, and the better
behaved the program.

Sandro

Loading...