+From mscott@sacadia.com Wed Nov 15 14:50:19 2000
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583
+ for <pgman@candle.pha.pa.us>; Wed, 15 Nov 2000 14:50:13 -0500 (EST)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998;
+ Wed, 15 Nov 2000 11:35:33 -0800 (PST)
+Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>,
+ Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us>
+Subject: Please help with some advice
+Message-ID: <Pine.GSO.4.10.10011151053260.9940-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Status: ORr
+
+Dear Sirs,
+
+I have been lurking on the PostgreSQL hackers list for about 3 months now
+and your names comes up more than any with helpful info about the project
+so I was hoping you could help me.
+
+Let me cut to the chase. I have been experimenting with 7.0.2 source to
+see if I could create a mutlti-threaded version of the backend so
+I could link directly from java ( I have a fe<->be protocol that I use for
+my apps). Needless to say I got into much more than I bargained for. I
+now have a version that works and it has some nice benefits that are very
+helpful to a project that I am working on. What I gained was
+
+prepared statements outside of spi
+batched commits (fsync)
+one connection per thread
+ multiple threads per process
+ multiple processes per installation
+
+I never really intended for anyone else to see the work so I drifted
+pretty far from the original code. I also ended up using Solaris threads
+rather than pthreads, I did my own implementation of the bufmgr.c and
+gram.y, and used Solaris implementation of mutex in place of S_LOCK and
+TAS. I grabbed all global variables and put them in an environment
+variable that is thread local. I also did some really stupid
+things like making TransactionId uint64 and making all my inserts use the
+same oid.
+
+My question is this. I would like to get some critical feedback and
+suggestions about the work from others. What is the best way to go about
+this? I thought about trying to create a project on greatbridge.org
+but I am rather new to open source and the code needs commented properly
+and cleaned up before too many try and look at it.
+
+Any suggestions would be greatly appreciated.
+
+
+Thanks in advance,
+
+Myron Scott
+
+
+
+From mscott@sacadia.com Thu Nov 16 17:19:45 2000
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315
+ for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:19:43 -0500 (EST)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449;
+ Thu, 16 Nov 2000 14:05:15 -0800 (PST)
+Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: Bruce Momjian <pgman@candle.pha.pa.us>
+cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
+Subject: Re: Please help with some advice
+In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us>
+Message-ID: <Pine.GSO.4.10.10011161401570.11441-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Status: OR
+
+Bruce Momjian wrote:
+
+>I am curious how you isolated each thread. It seems we pretty much
+>assume all our memory is controlled by a single query in the process.
+
+
+I moved all global variables to a thread global variable which is accessed
+by the method GetEnv(). Which looks like this
+
+Env* GetEnv(void) {
+ Env* env;
+ thr_getspecific(*envkey,(void*)&env);
+ return env;
+}
+
+The Env struct includes the CurrentMemoryContext, TopMemoryContext,
+PortalHeapMemory for each instance of a connection (one thread per
+connection). So, for example,
+EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
+
+void
+EndPortalAllocMode()
+{
+ PortalHeapMemory context;
+
+ AssertState(PortalManagerEnabled);
+ AssertState(IsA(GetEnv()->CurrentMemoryContext,
+PortalHeapMemory));
+
+ context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
+ AssertState(PointerIsValid(context->block)); /* XXX
+Trap(...) */
+
+ /* free current mode */
+ AllocSetReset(&HEAPMEMBLOCK(context)->setData);
+ MemoryContextFree((MemoryContext)
+PortalHeapMemoryGetVariableMemory(context),
+ context->block);
+
+ /* restore previous mode */
+ context->block = FixedStackPop(&context->stackData);
+}
+
+
+
+
+From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000
+Received: from sectorbase2.sectorbase.com ([208.48.122.131])
+ by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562
+ for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:23:21 -0500 (EST)
+Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21)
+ id <V8XQB5RW>; Thu, 16 Nov 2000 14:05:24 -0800
+Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com>
+From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
+To: "'Myron Scott'" <mscott@sacadia.com>,
+ Bruce Momjian
+ <pgman@candle.pha.pa.us>
+Cc: Tom Lane <tgl@sss.pgh.pa.us>
+Subject: RE: Please help with some advice
+Date: Thu, 16 Nov 2000 14:09:30 -0800
+MIME-Version: 1.0
+X-Mailer: Internet Mail Service (5.5.2650.21)
+Content-Type: text/plain;
+ charset="iso-8859-1"
+Status: ORr
+
+I think the question do we want to make backend multy-threaded
+should be discussed in hackers.
+
+Vadim
+
+> -----Original Message-----
+> From: Myron Scott [mailto:mscott@sacadia.com]
+> Sent: Thursday, November 16, 2000 2:05 PM
+> To: Bruce Momjian
+> Cc: Mikheev, Vadim; Tom Lane
+> Subject: Re: Please help with some advice
+>
+>
+> Bruce Momjian wrote:
+>
+> >I am curious how you isolated each thread. It seems we pretty much
+> >assume all our memory is controlled by a single query in the process.
+>
+>
+>
+> I moved all global variables to a thread global variable
+> which is accessed
+> by the method GetEnv(). Which looks like this
+>
+> Env* GetEnv(void) {
+> Env* env;
+> thr_getspecific(*envkey,(void*)&env);
+> return env;
+> }
+>
+> The Env struct includes the CurrentMemoryContext, TopMemoryContext,
+> PortalHeapMemory for each instance of a connection (one thread per
+> connection). So, for example,
+> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
+>
+> void
+> EndPortalAllocMode()
+> {
+> PortalHeapMemory context;
+>
+> AssertState(PortalManagerEnabled);
+> AssertState(IsA(GetEnv()->CurrentMemoryContext,
+> PortalHeapMemory));
+>
+> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
+> AssertState(PointerIsValid(context->block)); /* XXX
+> Trap(...) */
+>
+> /* free current mode */
+> AllocSetReset(&HEAPMEMBLOCK(context)->setData);
+> MemoryContextFree((MemoryContext)
+> PortalHeapMemoryGetVariableMemory(context),
+> context->block);
+>
+> /* restore previous mode */
+> context->block = FixedStackPop(&context->stackData);
+> }
+>
+>
+>
+
+From mscott@sacadia.com Thu Nov 16 22:16:38 2000
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638
+ for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 22:16:36 -0500 (EST)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874;
+ Thu, 16 Nov 2000 19:04:48 -0800 (PST)
+Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: Bruce Momjian <pgman@candle.pha.pa.us>
+cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
+Subject: Re: Please help with some advice
+In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us>
+Message-ID: <Pine.GSO.4.10.10011161904140.11870-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Status: ORr
+
+Thanks very much, I will post to hackers.
+
+Myron
+
+
+
+From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195
+ for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 00:30:19 -0500 (EST)
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335;
+ Tue, 2 Jan 2001 00:30:45 -0500 (EST)
+ (envelope-from pgsql-hackers-owner+M2691@postgresql.org)
+Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged))
+ by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232
+ for <pgsql-hackers@postgresql.org>; Tue, 2 Jan 2001 00:30:32 -0500 (EST)
+ (envelope-from mscott@sacadia.com)
+Received: from [206.112.108.0] (HELO sacadia.com)
+ by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2)
+ with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500
+Message-ID: <3A5167DB.3050807@sacadia.com>
+Date: Mon, 01 Jan 2001 21:32:11 -0800
+From: Myron Scott <mscott@sacadia.com>
+Reply-To: mscott@sacadia.com
+User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0
+X-Accept-Language: en
+MIME-Version: 1.0
+To: "Ross J. Reedstrom" <reedstrm@rice.edu>
+CC: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Using Threads?
+References: <004401c058fd$fd498d40$f2356880@tracy> <Pine.GSO.4.10.10012032351040.28161-100000@goldengate.kojoworldwide.com.> <20001204113307.B5871@rice.edu>
+Content-Type: text/plain; charset=us-ascii; format=flowed
+Content-Transfer-Encoding: 7bit
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+For anyone interested,
+
+I have posted my multi-threaded version of PostgreSQL here.
+
+http://www.sacadia.com/mtpg.html
+
+It is based on 7.0.2 and the TAO CORBA ORB which is here.
+
+http://www.cs.wustl.edu/~schmidt/TAO.html
+
+Myron Scott
+mkscott@sacadia.com
+
+
+
+From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001
+Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169
+ for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 03:02:27 -0500 (EST)
+Received: (from bright@localhost)
+ by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623;
+ Tue, 2 Jan 2001 00:02:31 -0800 (PST)
+Date: Tue, 2 Jan 2001 00:02:31 -0800
+From: Alfred Perlstein <bright@wintelcom.net>
+To: Bruce Momjian <pgman@candle.pha.pa.us>
+Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten
+Message-ID: <20010102000230.C19572@fw.wintelcom.net>
+References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us>
+Mime-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+Content-Disposition: inline
+User-Agent: Mutt/1.2.5i
+In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500
+Status: OR
+
+* Bruce Momjian <pgman@candle.pha.pa.us> [010101 23:59] wrote:
+> > Alfred Perlstein <bright@wintelcom.net> writes:
+> > > One trick that may help is calling sched_yield(2) on a lock miss,
+> > > it's a POSIX call and quite new so you'd need a 'configure' test
+> > > for it.
+> >
+> > The author of the current s_lock code seems to have thought that
+> > select() with a zero delay would do the equivalent of sched_yield().
+> > I'm not sure if that's true on very many kernels, if indeed any...
+> >
+> > I doubt we could buy much by depending on sched_yield(); if you want
+> > to assume POSIX facilities, ISTM you might as well go for user-space
+> > semaphores and forget the whole TAS mechanism.
+>
+>
+> Another issue is that sched_yield brings in the pthreads library/hooks
+> on some OS's, which we certainly want to avoid.
+
+I know it's a major undertaking, but since the work is sort of done,
+have you guys considered the port to solaris threads and seeing about
+making a pthreads port of that?
+
+I know it would probably get you considerable gains under Windows
+at the expense of dropping some really really legacy system.
+
+Or you could do what apache (is rumored) does and have it do either
+threads or processes or both...
+
+--
+-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
+"I have the heart of a child; I keep it in a jar on my desk."
+
+From pgsql-hackers-owner+M4275@postgresql.org Mon Feb 5 21:45:00 2001
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA09262
+ for <pgman@candle.pha.pa.us>; Mon, 5 Feb 2001 21:44:59 -0500 (EST)
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f162ixx00920;
+ Mon, 5 Feb 2001 21:44:59 -0500 (EST)
+ (envelope-from pgsql-hackers-owner+M4275@postgresql.org)
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f162fSx00595
+ for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 21:41:29 -0500 (EST)
+ (envelope-from mscott@sacadia.com)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id SAA03298
+ for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 18:25:05 -0800 (PST)
+Date: Mon, 5 Feb 2001 18:25:05 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Using Threads?
+Message-ID: <Pine.GSO.4.10.10102051823210.3289-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+I have put a new version of my multi-threaded
+postgresql experiment at
+
+http://www.sacadia.com/mtpg.html
+
+This one actually works. I have added a server
+based on omniORB, a CORBA 2.3 ORB from ATT. It
+ is much smaller than TAO and uses the thread per
+connection model. I haven't added the java side
+of the JNI interface yet but the C++ side is there.
+
+It's still not stable but it is much better than
+the last.
+
+Myron Scott
+mkscott@sacadia.com
+
+
+
+
+
+From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
+ for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
+ Tue, 6 Feb 2001 10:24:11 -0500 (EST)
+ (envelope-from pgsql-hackers-owner+M4304@postgresql.org)
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
+ for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
+ (envelope-from mscott@sacadia.com)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
+ Tue, 6 Feb 2001 07:05:04 -0800 (PST)
+Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: Karel Zak <zakkr@zf.jcu.cz>
+cc: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Using Threads
+In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
+Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+
+>
+> Sorry I haven't time to see and test your experiment,
+> but I have a question. How you solve memory management?
+> The current mmgr is based on global variable
+> CurrentMemoryContext that is very often changed and used.
+> Use you for this locks? If yes it is probably problematic
+> point for perfomance.
+>
+> Karel
+>
+
+There are many many globals I had to work around including all the memory
+management stuff. I basically threw everything into and "environment"
+variable which I stored in a thread specific using thr_setspecific.
+
+Performance is acually very good for what I am doing. I was able to batch
+commit transactions which cuts down on fsync calls, use prepared
+statements from my client using CORBA, and the various locking calls for
+the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
+some performance tests for inserts
+
+20 clients, 900 inserts per client, 1 insert per transaction, 4 different
+tables.
+
+7.0.2 About 10:52 average completion
+multi-threaded 2:42 average completion
+7.1beta3 1:13 average completion
+
+If I increased the number of inserts per transaction, multi-threaded got
+closer to 7.1 for inserts. I haven't tested other other types of
+commands
+yet.
+
+
+Myron Scott
+mkscott@sacadia.com
+
+
+From pgsql-hackers-owner+M4313@postgresql.org Tue Feb 6 12:32:00 2001
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA29163
+ for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 12:31:59 -0500 (EST)
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16HVox17454;
+ Tue, 6 Feb 2001 12:31:51 -0500 (EST)
+ (envelope-from pgsql-hackers-owner+M4313@postgresql.org)
+Received: from ara.zf.jcu.cz (ara.zf.jcu.cz [160.217.161.4])
+ by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16HV6x17323
+ for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 12:31:06 -0500 (EST)
+ (envelope-from zakkr@zf.jcu.cz)
+Received: from localhost (zakkr@localhost)
+ by ara.zf.jcu.cz (8.9.3/8.9.3/Debian 8.9.3-21) with SMTP id SAA03980;
+ Tue, 6 Feb 2001 18:31:02 +0100
+Date: Tue, 6 Feb 2001 18:31:02 +0100 (CET)
+From: Karel Zak <zakkr@zf.jcu.cz>
+To: Myron Scott <mscott@sacadia.com>
+cc: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Using Threads
+In-Reply-To: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
+Message-ID: <Pine.LNX.3.96.1010206182112.3799B-100000@ara.zf.jcu.cz>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+
+On Tue, 6 Feb 2001, Myron Scott wrote:
+
+> There are many many globals I had to work around including all the memory
+> management stuff. I basically threw everything into and "environment"
+> variable which I stored in a thread specific using thr_setspecific.
+
+ Yes, it's good. I working on multi-thread application server
+(http://mape.jcu.cz) and I use for this project some things from PG (like
+mmgr), I planning use same solution.
+
+> Performance is acually very good for what I am doing. I was able to batch
+> commit transactions which cuts down on fsync calls, use prepared
+> statements from my client using CORBA, and the various locking calls for
+> the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
+> some performance tests for inserts
+>
+> 20 clients, 900 inserts per client, 1 insert per transaction, 4 different
+> tables.
+>
+> 7.0.2 About 10:52 average completion
+> multi-threaded 2:42 average completion
+> 7.1beta3 1:13 average completion
+
+It is very very good for time for 7.1, already look forward to 7.2! :-)
+
+ BTW, I not sure if you anytime in future will see threads in
+official PostgreSQL and if you spending time on relevant things (IMHO).
+
+ Karel
+
+
+
+
+
+
+From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
+ for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
+Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
+ Tue, 6 Feb 2001 10:24:11 -0500 (EST)
+ (envelope-from pgsql-hackers-owner+M4304@postgresql.org)
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
+ for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
+ (envelope-from mscott@sacadia.com)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
+ Tue, 6 Feb 2001 07:05:04 -0800 (PST)
+Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: Karel Zak <zakkr@zf.jcu.cz>
+cc: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Using Threads
+In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
+Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+
+>
+> Sorry I haven't time to see and test your experiment,
+> but I have a question. How you solve memory management?
+> The current mmgr is based on global variable
+> CurrentMemoryContext that is very often changed and used.
+> Use you for this locks? If yes it is probably problematic
+> point for perfomance.
+>
+> Karel
+>
+
+There are many many globals I had to work around including all the memory
+management stuff. I basically threw everything into and "environment"
+variable which I stored in a thread specific using thr_setspecific.
+
+Performance is acually very good for what I am doing. I was able to batch
+commit transactions which cuts down on fsync calls, use prepared
+statements from my client using CORBA, and the various locking calls for
+the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
+some performance tests for inserts
+
+20 clients, 900 inserts per client, 1 insert per transaction, 4 different
+tables.
+
+7.0.2 About 10:52 average completion
+multi-threaded 2:42 average completion
+7.1beta3 1:13 average completion
+
+If I increased the number of inserts per transaction, multi-threaded got
+closer to 7.1 for inserts. I haven't tested other other types of
+commands
+yet.
+
+
+Myron Scott
+mkscott@sacadia.com
+
+
+From lamar.owen@wgcr.org Thu Jun 28 11:14:10 2001
+Return-path: <lamar.owen@wgcr.org>
+Received: from www.wgcr.org (IDENT:root@www.wgcr.org [206.74.232.194])
+ by candle.pha.pa.us (8.10.1/8.10.1) with ESMTP id f5SFE9U18758
+ for <pgman@candle.pha.pa.us>; Thu, 28 Jun 2001 11:14:09 -0400 (EDT)
+Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3])
+ by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id LAA11879;
+ Thu, 28 Jun 2001 11:14:14 -0400
+Content-Type: text/plain;
+ charset="iso-8859-1"
+From: Lamar Owen <lamar.owen@wgcr.org>
+To: Bruce Momjian <pgman@candle.pha.pa.us>
+Subject: Process weight (was:Re: [GENERAL] Re: Red Hat to support PostgreSQL)
+Date: Thu, 28 Jun 2001 11:14:09 -0400
+X-Mailer: KMail [version 1.2]
+References: <200106272258.f5RMwIb26959@candle.pha.pa.us>
+In-Reply-To: <200106272258.f5RMwIb26959@candle.pha.pa.us>
+MIME-Version: 1.0
+Message-ID: <01062811140902.01118@lowen.wgcr.org>
+Content-Transfer-Encoding: 8bit
+Status: ORr
+
+On Wednesday 27 June 2001 18:58, Bruce Momjian wrote:
+> > I had almost given up on using Postgres for this system because under
+> > Solaris, it just couldn't cut it (MySQL could do the work with one CPU
+> > while Postgres took up even more CPU and required *both* CPUs to be
+> > enabled), but when we moved the system to a Linux box, things worked
+> > much better.
+
+> Ah, back to a PostgreSQL topic. :-)
+
+> My guess on this one is that Solaris is slower for PostgreSQL because
+> process switching is _much_ heavier on Solaris than other OS's. This is
+> because of the way they implemented processes in SVr4. They got quite
+> heavy, almost requiring kernel threads so you weren't switching
+> processes all the time.
+
+Now, the question of the week:
+Is supporting a thread model for an inefficient OS a desirable thing to do,
+when more efficient OS kernels are available such as FreeBSD 4.x and Linux
+2.4? My opinion is that our existing model, when used with a
+connection-pooling frontend, is rather efficient. (Yes, I use a
+connection-pooling frontend. Performance is rather nice, and I don't have to
+have a full backend spawned for every page hit.)
+
+In fact, on a Linux box threads show as processes. While I know that the
+kernel actually supports themin a slightly different manner than processes,
+they have more similarities than differences.
+
+However, even on OS's where threads are supported, the mechanism to support
+those threads must be an efficient one -- not all pthreads libraries are
+created equal. Many are frontends (expensive ones, at that) for plain old
+processes.
+
+Does anyone know of a resource that details the 'weight' of processes for our
+supported platforms? [reply off-list -- I'll be glad to summarize responses
+to HACKERS, ADMIN, or PORTS, as appropriate, if desired.]
+--
+Lamar Owen
+WGCR Internet Radio
+1 Peter 4:11
+
From pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:25:32 2001
Return-path: <pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org>
Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
+From pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:40:26 2001
+Return-path: <pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org>
+Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
+ by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMePo13437
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 18:40:25 -0400 (EDT)
+Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMeZ417944
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:40:35 -0500 (CDT)
+ (envelope-from pgsql-hackers-owner+M13604=candle.pha.pa.us=pgman@postgresql.org)
+Received: from foghorn.airs.com (foghorn.airs.com [63.201.54.26])
+ by postgresql.org (8.11.3/8.11.4) with SMTP id f8QM59h01247
+ for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 18:05:09 -0400 (EDT)
+ (envelope-from ian@airs.com)
+Received: (qmail 10089 invoked by uid 10); 26 Sep 2001 22:04:49 -0000
+Received: (qmail 6837 invoked by uid 269); 26 Sep 2001 22:04:41 -0000
+Mail-Followup-To: markw@mohawksoft.com,
+ pgsql-hackers@postgresql.org,
+ dhageman@dracken.com
+To: "D. Hageman" <dhageman@dracken.com>
+cc: mlw <markw@mohawksoft.com>,
+ "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
+Subject: Re: [HACKERS] Spinlock performance improvement proposal
+References: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
+From: Ian Lance Taylor <ian@airs.com>
+Date: 26 Sep 2001 15:04:41 -0700
+In-Reply-To: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
+Message-ID: <si8zf1vcau.fsf@daffy.airs.com>
+Lines: 45
+User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7
+MIME-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+"D. Hageman" <dhageman@dracken.com> writes:
+
+> > you have a newer kernel scheduled implementation, then you will have the same
+> > scheduling as separate processes. The only thing you will need to do is
+> > switch your brain from figuring out how to share data, to trying to figure
+> > out how to isolate data. A multithreaded implementation lacks many of the
+> > benefits and robustness of a multiprocess implementation.
+>
+> Save for the fact that the kernel can switch between threads faster then
+> it can switch processes considering threads share the same address space,
+> stack, code, etc. If need be sharing the data between threads is much
+> easier then sharing between processes.
+
+When using a kernel threading model, it's not obvious to me that the
+kernel will switch between threads much faster than it will switch
+between processes. As far as I can see, the only potential savings is
+not reloading the pointers to the page tables. That is not nothing,
+but it is also not a lot.
+
+> I can't comment on the "isolate data" line. I am still trying to figure
+> that one out.
+
+Sometimes you need data which is specific to a particular thread.
+Basically, you have to look at every global variable in the Postgres
+backend, and determine whether to share it among all threads or to
+make it thread-specific. In other words, you have to take extra steps
+to isolate the data within the thread. This is the reverse of the
+current situation, in which you have to take extra steps to share data
+among all backend processes.
+
+> That last line is a troll if I every saw it ;-) I will agree that threads
+> isn't for everything and that it has costs just like everything else. Let
+> me stress that last part - like everything else. Certain costs exist in
+> the present model, nothing is - how should we say ... perfect.
+
+When writing in C, threading inevitably loses robustness. Erratic
+behaviour by one thread, perhaps in a user defined function, can
+subtly corrupt the entire system, rather than just that thread. Part
+of defensive programming is building barriers between different parts
+of a system. Process boundaries are a powerful barrier.
+
+(Actually, though, Postgres is already vulnerable to erratic behaviour
+because any backend process can corrupt the shared buffer pool.)
+
+Ian
+
+---------------------------(end of broadcast)---------------------------
+TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
+
+From pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 18:54:58 2001
+Return-path: <pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org>
+Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
+ by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QMsvo14061
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 18:54:57 -0400 (EDT)
+Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QMt7420740
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:55:07 -0500 (CDT)
+ (envelope-from pgsql-hackers-owner+M13605=candle.pha.pa.us=pgman@postgresql.org)
+Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
+ by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QMOPh04333
+ for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 18:24:26 -0400 (EDT)
+ (envelope-from mscott@sacadia.com)
+Received: from localhost (localhost [127.0.0.1])
+ by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id PAA00633
+ for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 15:03:00 -0700 (PDT)
+Date: Wed, 26 Sep 2001 15:03:00 -0700 (PDT)
+From: Myron Scott <mscott@sacadia.com>
+X-Sender: mscott@goldengate.kojoworldwide.com.
+To: pgsql-hackers@postgresql.org
+Subject: Re: [HACKERS] Spinlock performance improvement proposal
+In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com>
+Message-ID: <Pine.GSO.4.10.10109261428340.563-100000@goldengate.kojoworldwide.com.>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: OR
+
+
+
+On Wed, 26 Sep 2001, mlw wrote:
+
+> I can only think of two objectives for threading. (1) running the various
+> connections in their own thread instead of their own process. (2) running
+> complex queries across multiple threads.
+>
+
+I did a multi-threaded version of 7.0.2 using Solaris threads about a year
+ago in order to try
+and get multiple backend connections working under one java process using
+jni. I used the thread per connection model.
+
+I eventually got it working, but it was/is very messy ( there were global
+variables everywhere! ). Anyway, I was able to get a pretty good speed up
+on inserts by scheduling buffer writes from multiple connections on one
+common writing thread.
+
+I also got some other features that were important to me at the time.
+
+1. True prepared statements under java with bound input and output
+variables
+2. Better system utilization
+ a. fewer Solaris lightweight processes mapped to threads.
+ b. Fewer open files per postgres installation
+3. Automatic vacuums when system activity is low by a daemon thread.
+
+but there were some drawbacks... One rogue thread or bad user
+function could take down all connections for that process. This
+was and seems to still be the major drawback to using threads.
+
+
+Myron Scott
+mscott@sacadia.com
+
+
+---------------------------(end of broadcast)---------------------------
+TIP 5: Have you checked our extensive FAQ?
+
+http://www.postgresql.org/users-lounge/docs/faq.html
+
+From pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:45:26 2001
+Return-path: <pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org>
+Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
+ by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLjQo08483
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:45:26 -0400 (EDT)
+Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
+ by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLjY409914
+ for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 16:45:35 -0500 (CDT)
+ (envelope-from pgsql-hackers-owner+M13602=candle.pha.pa.us=pgman@postgresql.org)
+Received: from typhon.dracken.com (dv07m61.lawrence.ks.us [24.124.61.35])
+ by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QLGDh91021
+ for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 17:16:13 -0400 (EDT)
+ (envelope-from dhageman@dracken.com)
+Received: from localhost (dhageman@localhost)
+ by typhon.dracken.com (8.11.4/8.11.4) with ESMTP id f8QLEMY01973;
+ Wed, 26 Sep 2001 16:14:22 -0500
+X-Authentication-Warning: typhon.dracken.com: dhageman owned process doing -bs
+Date: Wed, 26 Sep 2001 16:14:22 -0500 (CDT)
+From: "D. Hageman" <dhageman@dracken.com>
+To: mlw <markw@mohawksoft.com>
+cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
+Subject: Re: [HACKERS] Spinlock performance improvement proposal
+In-Reply-To: <3BB23DD6.E86AF327@mohawksoft.com>
+Message-ID: <Pine.LNX.4.33.0109261600100.1784-100000@typhon.dracken.com>
+MIME-Version: 1.0
+Content-Type: TEXT/PLAIN; charset=US-ASCII
+Precedence: bulk
+Sender: pgsql-hackers-owner@postgresql.org
+Status: ORr
+
+On Wed, 26 Sep 2001, mlw wrote:
+>
+> I can only think of two objectives for threading. (1) running the various
+> connections in their own thread instead of their own process. (2) running
+> complex queries across multiple threads.
+>
+> For item (1) I see no value to this. It is a lot of work with no tangible
+> benefit. If you have an old fashion pthreads implementation, it will hurt
+> performance because are scheduled within the single process's time slice..
+
+Old fashion ... as in a userland library that implements POSIX threads?
+Well, I would agree. However, most *modern* implementations are done in
+the kernel or kernel and userland coop model and don't have this
+limitation (as you mention later in your e-mail). You have kinda hit on
+one of my gripes about computers in general. At what point in time does
+one say something is obsolete or too old to support anymore - that it
+hinders progress instead of adding a "feature"?
+
+> you have a newer kernel scheduled implementation, then you will have the same
+> scheduling as separate processes. The only thing you will need to do is
+> switch your brain from figuring out how to share data, to trying to figure
+> out how to isolate data. A multithreaded implementation lacks many of the
+> benefits and robustness of a multiprocess implementation.
+
+Save for the fact that the kernel can switch between threads faster then
+it can switch processes considering threads share the same address space,
+stack, code, etc. If need be sharing the data between threads is much
+easier then sharing between processes.
+
+I can't comment on the "isolate data" line. I am still trying to figure
+that one out.
+
+That last line is a troll if I every saw it ;-) I will agree that threads
+isn't for everything and that it has costs just like everything else. Let
+me stress that last part - like everything else. Certain costs exist in
+the present model, nothing is - how should we say ... perfect.
+
+> For item (2) I can see how that could speed up queries in a low utilization
+> system, and that would be cool, but in a server that is under load, threading
+> the queries probably be less efficient.
+
+Well, I don't follow your logic and you didn't give any substance to back
+up your claim. I am willing to listen.
+
+Another thought ... Oracle uses threads doesn't it or at least it has a
+single processor and multi-processor version last time I knew ... which do
+they claim is better? (Not saying that Oracle's proclimation of what is
+good and what is not matters, but it is good for another view point).
+
+--
+//========================================================\\
+|| D. Hageman <dhageman@dracken.com> ||
+\\========================================================//
+
+
+---------------------------(end of broadcast)---------------------------
+TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
+
projects / postgresql / commitdiff