Input stream questions 
Author Message
 Input stream questions

> I am posting this proposal (which was passed at the most recent
> meeting of X3J14, because everything was deleted from it except for
> (16)) so that anyone and everyone can respond to the questions raised
> in it.

Another way of looking at this is that the proposal itself was not
actionable (i.e. there is no specific "Proposal" section, and
"Discussion" is not in itself actionable).

Nevertheless, we tried very hard to give it the "benefit of the doubt".
I read the discussion section at the meeting (it arrived in the middle
of the meeting, and I didn't have much time), found the one issue that
appeared to represent a real technical problem, and created an
actionable proposal out of that issue.

There are some things that could indeed use additional clarification,
although I believe that the letter of the Basis document is correct.
For those issues, we solicit proposals for specific improved wording.

Also, having had time to carefully reconsider all these points, I believe
that there is one more technical issue that needs to be addressed (see
the end of item 5.)  I don't think this is a serious problem, though;
rather, it is a usage restriction that should be noted.

> One member of the X3J14 Technical Committee (TC) has agreed to
> respond point-by-point to these questions.  Those responses will also
> be posted.

I'll shorten the loop by posting the response directly.  Read on.

> ----------------------------------------------------------------------
> ANSI ASC X3 / X3J14 Forth Technical Proposal    TP91-1082
> ----------------------------------------------------------------------
> Title:  Clarify BASIS on the subject of the Forth input stream.
> ----------------------------------------------------------------------
> Related Proposals:      TP90-833, TP91-1070
> ----------------------------------------------------------------------
> Keyword(s):     input stream, block file, implementation-defined
> default block space, text file
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> Abstract:       Clarify BASIS on the subject of the Forth input stream.
>  BASIS is still unclear on the specific input stream ``entitlements''
> to the point that we have heard differing interprpreter, 5.3.2
> addressable memory, Input Stream, 8.1.006 #TIB, 8.1.0080 (,
> 8.1.0560 >IN, 8.1.0790 BLK, 8.1.1360 EVALUATE, 10.1.1713 INCLUDE,
> 10.1.1717 INCLUDE-FILE, 11.1.1790 LOAD, 10.2.1792 LOAD-FILE, 8.2.2040
> QUERY, 8.2.2125 REFILL, 10.1.2218 SOURCE-FILE, 8.2.2240 SPAN, 8.1.2290
> TIB, and 8.2.2535 \ (plus other new sections).
> ----------------------------------------------------------------------
> Discussion:

> In the following discussion the term ``legitimate'' means ``portable,
> consistent with the intention of X3J14, and within the boundaries of
> good taste.''

> 1)      There is a question about the general nature of #TIB.  Which of
> the following are legitimate and correct in ANS Forth?

>        It appears, given the letter of BASIS, that the first ought to
> be correct in all cases, but is #TIB also meant to hold the input stream
> length?

The first is indeed correct.  #TIB is only valid if BLK is 0.  Reference: Input Stream (Basis 14).

> 2)      There is a question about the general nature of TIB.  Which of
> the following are legitimate and correct in ANS Forth?

>   THEN ;

>         It appears, given the letter of BASIS, that the first ought to
> be correct in all cases, but is TIB also meant to specify the beginning
> of the input stream (i.e., does it call BLOCK)?  Also, does the value
> returned by TIB ever change, e.g. during EVALUATE, INCLUDE, or
> INCLUDE-FILE, or do the values of #TIB and >IN change relative to it?

The first is correct.  TIB and BLOCK are completely unrelated.  The
value of TIB may or may not change, at the discretion of the implementor.
For example, a valid implementation could have a fixed TIB buffer, and save
its contents somewhere else when the input stream nests to a text file
or to an EVALUATEd string.  Another valid implementation could change
the address returned by TIB whenver the input stream nests.

The second implementation is more likely, because the standard imposes
no restriction on the length of the string that can be EVALUATEd, thus
in the first implementation, the fixed TIB buffer would have to be
arbitrarily large.

Consequently, a standard program should be written so it doesn't care
whether or not TIB changes across input stream boundaries.  This should
not be a problem; just use TIB anytime you need that address, rather
than saving a copy of its value somewhere.

Reference: Basis 14 does not say or imply that TIB is a fixed buffer.

> 3)      There is a question about what >IN holds.  Are the following
> legitimate and correct in ANS Forth?

> : initialize-stream  ( -- )     : skip-stream  ( -- )

>         That is, can anything but a previously saved value of >IN be
> legitimately stored in >IN by a user?

Both are legitimate and correct.  This used to not be the case, but
it was changed as part of the text file input stream stuff about a
year ago (there were some editorial errors that prevented it from
being correctly presented in the first couple of Basis documents
after the change was approved).

The confusion arose because of a rationale box below the glossary entry
for >IN .  That glossary entry is incorrect, and will not appear in
Basis 15 (Reference: proposal TP-1035).

Also, the entitlements for setting >IN are clarified in new wording
for section, as a result of proposal TP-1032.


> 4)      There is a question about what #TIB holds.  Is the following
> legitimate and correct in ANS Forth?

> : truncate-stream  ( -- )

>         That is, can #TIB ever be legitimately modified directly by a
> user?

No and no.  Basis is not crystal clear on this prohibition.  Specific
wording to make this clear would be appreciated.  TIB and #TIB should
be considered "read only" for user programs.

The paragraph in that seems to imply that #TIB is alterable
has been replaced.

> 5)      There is a question about what BLK holds.  Is the following
> legitimate and correct in ANS Forth?

> : ans-query  ( -- )
>   TIB +n ACCEPT  #TIB !  0 BLK !  0 >IN ! ;

No, because
  1) You don't necessarily know the size of the buffer that the current
     value of TIB points to.
  2) Storing into #TIB is bogus (the system can do it, but not the user).

>         That is, (in addition to the above questions) can BLK ever be
> legitimately modified directly by a user?

Yes, BLK can be modified.  The new wording for section (alluded-to
above) specifically says so.

>  If not, what does it mean to
> say that ``QUERY is semantically equivalent to [that] sequence...''
> (BASIS section 8.2.2040 QUERY)?

It means that the results of executing QUERY are as described by that
code.   It doesn't mean that the user is allowed to write that code
or similar code.

QUERY really ought to be described in English instead of in code.

Actually, QUERY is somewhat bogus; suppose you write and execute:

        : FOO " QUERY" EVALUATE  ;

This could overwrite the definition FOO !

This is worth a specific proposal; If I had noticed this, I would
have brought it up at the last meeting (I didn't have much time
to evaluate this proposal/discussion as it came in fairly late
in the meeting).

- Show quoted text -

> 6)      There is a question about accessing the contents of the input
> stream.  Is any of the following legitimate and correct in ANS Forth?

> : uppercase  ( c-addr u -- )  ... ;  ( converts string to uppercase )
> : uppercase-next-word  ( -- )

> : remaining-stream  ( c-addr u -- )

> : CHAR  ( "ccc" -- char )


>         Given that uppercase-next-word will not generally work if the
> input stream is coming from a block (and what about EVALUATE?), can the
> contents of the input stream ever be legitimately modified directly by a
> user (uppercase-next-word, above)?  Can the contents of the input stream
> ever be legitimately manipulated in any way directly by a user (CHAR,
> above), or is it only ever legitimate to parse from the input stream?

The remaining-stream / CHAR example is fine, but the uppercase-next-word
example has problems.  For instance, suppose the input stream is coming
from a string in read-only memory
        e.g. : FOO " UPPERCASE-NEXT-WORD xyzzy" EVALUATE  ;
where FOO has been precompiled into ROM.

> If it is not legitimate for a user to directly calculate addresses
> within the input stream without parsing, can a user ever do anything
> legitimate with the value of  TIB?

at that address, but writing into the input stream is not guaranteed to

Not that the CHAR example, although legitimate and correct, does not
skip leading delimiters, so it may or may not be what you want.

> 7)      There is a question about parsing.  Is the following legitimate
> and correct in ANS Forth?

> : CHAR  ( "ccc" -- char )

No problem.  Totally okay.

>         Section 4.0575 Parsing says, ``If the current input stream is
> empty or contains no characters other than the delimiter, the string is
> empty.''  Is an empty string-implementation defined, or is the count
> zero?

The count is zero.  This was unclear in the definitions of WORD (8.1.2450)
and PARSE (8.2.2008) and we passed proposal TP-1025 to fix it.

> 8)      Is it true that, ``words using TIB and modifying the contents
> of BLK, >IN, or #TIB are responsible for maintaining the integrity of
> the input stream specification and its contents''  (changes added)?
> Ought section Input Stream paragraph four (from where the above
> was taken) to be so modified?  Ought there to be more prose in that
> section about nesting and un-nesting the input stream and if so, what
> prose?

Proposal TP-1032 totally rewrote the section in question, with the
intention of making it clear what you can and cannot do to the input
stream.  I think the new prose is clear (John Hayes wrote it).  Your
mileage may vary.

> 9)      There is a question about CURRENT-FILE and SOURCE-FILE.  We
> have heard from the TC that CURRENT-FILE is obsolete, that SOURCE-FILE
> is obsolete, and that they are the same!  We most definitely believe
> (and assume) that CURRENT-FILE is not obsolete, that SOURCE-FILE (as a
> named value) is obsolete, and that they are definitely not the same
> thing (see TP91-1070).  C'mon, which is it guys?

The whole block-file/block/text-file thing has been cleaned up and
clarified.  Several of us stayed up until 2AM one night to make sure
everybody is on the same wavelength.  As a result, we passed TP-1070
(by the same authors as TP-1082, on which I am commenting), so that:

     BLOCK-FID is a variable containing the fileid of the file containing
     the block space, or containing 0 for the implementation-defined
     default block space.  (This used to called CURRENT-FILE but that
     name confused everybody.)  It is possible to store into this variable.

     SOURCE-FILE returns the fileid of the file that is the source of
     text file input, or 0 if the input is coming from the keyboard, or
     -1 if the input is coming from an EVALUATEd string.  SOURCE-FILE
     may not be directly modified by the user; it is set by the system
     as a result of executing INCLUDE-FILE (and related words) or

     BLOCK-FID determines the source of blocks, regardless of the value
     of SOURCE-FILE, and SOURCE-FILE determines the source of text
     input line, regardless of the value contained in BLOCK-FID .

> 10)     The wording of BASIS section 8.2.2125 REFILL is unclear and /
> or incorrect.  When BLK contains a non-zero value and SOURCE-FILE
> returns a non-zero value, section 8.2.2125 specifies that REFILL takes
> the action ``0 >IN !  1 BLK +! 0.''  First, does that means that the
> system next interprets the block specified by the new value in BLK in
> the ``block space'' specified by CURRENT-FILE (regardless of the value
> in SOURCE-FILE)?  Second, if CURRENT-FILE contains a non-zero value,
> shouldn't REFILL return other than zero, or should the system simply
> abort if a user attempts a REFILL beyond the end of a block file?

This was indeed a problem.  TP-1048 fixed it.  Thanks for bringing
up the issue.  REFILL can now return "sorry, didn't work" for blocks.

> 11)     Is SOURCE-FILE useful in any way to a user?  It is used in
> BASIS to help explain the way INCLUDE, INCLUDE-FILE, and REFILL work,
> but it can only be modified by INCLUDE, INCLUDE-FILE and little can be
> done with it by a user except defining:  : INTERPRETING-TEXT-FILE?  ( --
> flag )  SOURCE-FILE 0<> ;.  We cannot see the need for that system value
> to be given a name and will help rewrite BASIS in terms of ``source
> file'' (the file whose text gets interpreted by INCLUDE and
> INCLUDE-FILE), removing any reference to SOURCE-FILE.

This observation is correct, and the example of how it may be useful
is also correct.  However, it is my belief that there is value in
exposing SOURCE-FILE to the user, because specific systems may choose
to provide other file operations in addition to the set of standard
ones.  It is nice to have a standard name for the word thats returns
the fileid of the input file; every text-file-based system is going
to have the word anyway, so they might as well use the same name!

> 12)     Section 10.1.1029 CURRENT-FILE refers to ``current block file''
> and ``implementation-defined default block space,'' neither of which are
> defined.  A ``block file'' probably means ``whenever the contents of
> CURRENT-FILE are non-zero'' and differs from the implementation-defined
> default block space only in that the concept of end-of-file is
> meaningful for a block file.  Ought ``block file,'' ``current block
> file,'' and ``implementation-defined default block space'' be defined in
> the Definition of Terms and if so, how?

Agreed (except that the name is now BLOCK-FID).  Specific wording would
be appreciated.

Actually, the concept of "end-of-file" may be meaningful for the
implementation-defined default block space for some implementations.

"implementation-defined default block space" is the place where blocks
come from if the program doesn't do anything to explicitly change it
(like storing a new fileid in BLOCK-FID, or executing LOAD-FILE).

> 13)     There are some unclear aspects of the specification of comments
> (BASIS sections 8.1.0080( and 8.2.2535 \).  Section 8.1.0080 refers to
> ``text file,'' but that term is nowhere defined.  It probably means
> ``whenever the contents of BLK are zero and the source fileid is
> non-zero.''  Ought ``text file'' be defined in the Definition of Terms?

Yes, it should.  Again, a proposal suggesting specific wording would
be appreciated.

> Section 8.2.235 refers to ``line,'' but section 4.0510 (the definition
> of the term ``line'') is inadequate to describe the lines in blocks (and
> block files) versus those in text files (it describes how ``lines'' are
> displayed).  What is meant by a line within a block (e.g. is it 64
> characters) and to what does section 4.0510 refer?  Ought the wording of
> section 4.0510 be improved and if so, how?

It was changed by TP-1021.  The current wording says something to the
effect that a line is what you see on one line of a display, and that
BLOCK source is conventionally represented as 16 lines of 64 characters.
I don't have the actual wording the we agree on written down; but it will
be in Basis 15.

> 14)     There are some questions about (.
>       - Section 8.1.0080 ( spells out that a null comment, (), is
> allowed so that ( cannot be implemented in terms of WORD (though
> sections 8.1.0190 ." and 8.2.0200 .( do not specify that behavior).
> Does that behavior need a rationale note?

Probably.  Please write one.  By the way, ( and ." and .( and S" and
C" can be implemented in terms of PARSE, which was one of the reasons
for including the word PARSE .

>       - Sections 8.1.0080 ( and 8.2.0200 .( specify that those words
> are immediate.  Is that a mistake and if not, why not?

What we really mean by this is that the compilation semantics and
the execution semantics of ( and .( are identical.

For non-immediate words, the compilation semantics is to add the
execution semantics to the current definition.

For most other immediate words, there are distinctly-different
compilation semantics and execution semantics.  For example,
the compilation semantics of IF involve pushing something on
the control flow stack and compiling a transfer-of-control, whereas
the execution semantics involve the data stack.

Considering the small number of words like ( , with identical
compilation and execution semantics, it would probably be a
good idea to just spell it out in the glossary entry.

Note, however, that this is a non-technical issue, in that the
clarification does not change the meaning.  Thus, such a change
could be made after the document goes to the dpANS review stage.

>       - Does X3J14 really want to require ( to comment to the end of a
> text file?

This point has been argued over and over, and we could never get an
overwhelming majority.  The vote was consistenly about 65% for multi-line
comments, 35% against (my surveys of users suggest a much stronger
majority in favor of multi-line comments).  The ultimate decision was
to put the issue to a letter ballot (to which ALL committee members,
not just the ones present at a particular meeting, MUST respond).
Then we would just go with a simple majority of that vote.  Everybody
at the meeting was willing to go along with that, because nobody felt
so strongly about the issue that they were willing to make a federal
case of it.

> 15)     Is the following legitimate and correct in ANS Forth and does
> it immediately exit the current input stream?

>   IF   1024 >IN !
>        BEGIN
>             REFILL 0=
>        UNTIL
>   THEN ;


> 16)     We asked the question of the Input Stream Working Group, ``why
> does section 11.1.1790 LOAD have the stack diagram ( i*w -- j*w )?'' and
> got explanations as to what the ``i*x'' mean.  To us, that stack comment
> says, ``LOAD takes a variable number of arguments and leaves a variable
> (and potentially different) number of arguments.''  Didn't LOAD used to
> take a block number?

The stack diagram should be  ( i*w u -- j*w ) , where "u" is the block
number.  A similar stack diagram applies to LOAD-FILE .  The i*w and
j*w implies that LOAD and LOAD-FILE may interpret code that has the
effect of popping, pushing, or modifying arbitrary stack items.

This discussion item was "promoted" from a discussion item to an actual

> ----------------------------------------------------------------------
> Date:           February 1, 1991
> Submitted by:   David C. Petty and Bent Schmidt-Nielsen
> Address:        Post Office Box Two
>                 Cambridge,  MA    02140-0001

> Telephone:      +1(617)492-1232         FAX:    +1(617)491-2345

> ANSI ASC X3 / X3J14 Forth Standards Committee
> 111 North Sepulveda Boulevard, Suite 300  Manhattan Beach,  CA
> 90266-6861
> -----------------------------------------------------------------------
> X3J14/87-021    12/04/1987

Thanks for bringing up these issues.

The following is a general request directed at all proposal writers:

It is *REALLY* helpful if proposals are broken down into smaller pieces.
The logistic difficulties of dealing with "omnibus" proposals are severe.
This hurts the chances of the proposal.

Also, proposals suggesting specific wording are much easier to handle
than proposals to the effect of "you guys should do something about this".
Basically, in order to act on the latter, the proposal has to go back
in the pile, and somebody has to volunteer to take ownership of the issue
and write up some specific wording that can be acted on.  Speaking from
personal experience as a person who has done more than my share of such
volunteering, it is a pain in the {*filter*}to do this, and the time it takes
to do it sometimes causes me to miss the discussion on other issues.

Sun, 01 Aug 1993 10:44:42 GMT  
 [ 1 post ] 

 Relevant Pages 

1. question about the fate of input stream after DATA_ERROR

2. Adding a line to the input stream

3. Changing the input file stream

4. GZip Input Stream - anywhere ???

5. Standard Input Stream

6. Conventions for the input stream

7. Input Stream

8. Input stream addressability

9. input-stream and WORD

10. non-referentially transparent input stream?

11. Input/output streams in Haskell and Miranda

12. CHAR's in ETH's input stream


Powered by phpBB® Forum Software